Exercise 6.4.4
Deadly triad and DQN: instability, experience replay, target network
Reinforcement Learning › Exercises › Lecture 6 · Function Approximation › 6.4 Convergence, divergence and experience replay
The Challenge of Stable Function Approximation methods
-
a) It is well-known that combining function approximation, bootstrapping (as in TD methods), and off-policy training can lead to instability and divergence. From a theoretical standpoint, explain why the interaction of these three elements is so problematic for learning.
-
b) The Deep Q-Network (DQN) algorithm is a prominent example of a stable off-policy method that uses a powerful non-linear function approximator. Analyze the key architectural and algorithmic components of DQN and explain how they specifically address the instability issues you described in part (a).