Exercise 7.5.1

Actor-critic: roles of actor and critic, TD error as baseline

Reinforcement Learning › Exercises › Lecture 7 · Policy Gradients and Actor-Critic › 7.5 Actor-critic, advantage function, A2C
EasyWritten

Actor-Critic methods can be seen as a natural evolution of the REINFORCE algorithm with a baseline, designed to reduce variance and improve learning stability.

  • a) Describe the architecture of an Actor-Critic agent. Clearly define the roles of the ‘actor’ and the ‘critic’ and explain what each component learns.

  • b) In the policy update for the actor, the critic’s output is used to form a learning signal. Explain the connection between this learning signal and the ‘(Return - Baseline)’ term in the REINFORCE algorithm. Why is the Actor-Critic approach generally considered an advancement over REINFORCE with a simple baseline?