Exercise 6.4.4

Deadly triad and DQN: instability, experience replay, target network

Reinforcement Learning › Exercises › Lecture 6 · Function Approximation › 6.4 Convergence, divergence and experience replay
MediumWritten

The Challenge of Stable Function Approximation methods

  • a) It is well-known that combining function approximation, bootstrapping (as in TD methods), and off-policy training can lead to instability and divergence. From a theoretical standpoint, explain why the interaction of these three elements is so problematic for learning.

  • b) The Deep Q-Network (DQN) algorithm is a prominent example of a stable off-policy method that uses a powerful non-linear function approximator. Analyze the key architectural and algorithmic components of DQN and explain how they specifically address the instability issues you described in part (a).