Robust Actor-Critic With Relative Entropy Regulating Actor

Authors：Cheng, Yuhu; Huang, Longyang; Chen, C. L. Philip; Wang, Xuesong^*

Source：IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(11): 9054-9063.

DOI：10.1109/TNNLS.2022.3155483

• Entropy
• Task analysis
• Convergence
• Standards
• Stability criteria
• Space exploration
• Q-learning
• Actor-critic
• reinforcement learning (RL)
• relative entropy
• robust policy improvement mechanism (RPIM)

The accurate estimation of Q-function and the enhancement of agent's exploration ability have always been challenges of off-policy actor-critic algorithms. To address the two concerns, a novel robust actor-critic (RAC) is developed in this article. We first derive a robust policy improvement mechanism (RPIM) by using the local optimal policy about the current estimated Q-function to guide policy improvement. By constraining the relative entropy between the new policy and the previous one in policy improvement, the proposed RPIM can enhance the stability of the policy update process. The theoretical analysis shows that the incentive to increase the policy entropy is endowed when the policy is updated, which is conducive to enhancing the exploration ability of agents. Then, RAC is developed by applying the proposed RPIM to regulate the actor improvement process. The developed RAC is proven to be convergent. Finally, the proposed RAC is evaluated on some continuous-action control tasks in the MuJoCo platform and the experimental results show that RAC outperforms several state-of-the-art reinforcement learning algorithms.