Exercise 5.5.2

Maximization bias: why Q-learning overestimates, mitigation

Reinforcement Learning › Exercises › Lecture 5 · Model-Free Control › 5.5 Maximization bias and double Q-learning
EasyWritten

Q-learning updates its Q-values with the following rule: \[Q(S_t, A_t) \leftarrow Q(S_t, A_t) + \alpha \left[R_{t+1} + \gamma \max_{a'} Q(S_{t+1}, a') - Q(S_t, A_t)\right].\] A known issue with Q-learning is its tendency to overestimate Q-values.

  • (a) Explain why Q-learning is susceptible to overestimation of Q-values.

  • (b) Additionally, consider the implications of this overestimation for the convergence of Q-learning and suggest a possible method to mitigate this issue.