Exercise 7.5.1
Actor-critic: roles of actor and critic, TD error as baseline
Reinforcement Learning › Exercises › Lecture 7 · Policy Gradients and Actor-Critic › 7.5 Actor-critic, advantage function, A2C
Actor-Critic methods can be seen as a natural evolution of the REINFORCE algorithm with a baseline, designed to reduce variance and improve learning stability.
-
a) Describe the architecture of an Actor-Critic agent. Clearly define the roles of the ‘actor’ and the ‘critic’ and explain what each component learns.
-
b) In the policy update for the actor, the critic’s output is used to form a learning signal. Explain the connection between this learning signal and the ‘(Return - Baseline)’ term in the REINFORCE algorithm. Why is the Actor-Critic approach generally considered an advancement over REINFORCE with a simple baseline?