Exercise 1.9.4
Softmax over Q* (SARSA) vs softmax over action preferences
This question explores the fundamental difference between creating a stochastic policy as a behavioral component of a value-based method versus learning it as the direct objective of an optimization process.
-
(a) Consider a value-based agent (e.g., using SARSA) that employs a softmax policy for action selection throughout its learning process. The policy is defined based on its current action-value estimates: \[\pi(a|s) = \frac{\exp(\hat{q}(s,a)/\tau)}{\sum_{b} \exp(\hat{q}(s,b)/\tau)}\] Now, assume the agent learns for a very long time, and its action-value estimates successfully converge to the true optimal values, \(\hat{q}(s,a) \to Q^*(s,a)\). Analyze the agent’s final, converged policy (assuming \(\tau=1\)). Does this policy represent the optimal policy? Explain why or why not by describing the action probabilities it will assign in a state with one optimal action and several suboptimal (but not terrible) actions.
-
(b) Now consider a policy gradient method that learns a set of "action preferences," \(h(s,a,\theta)\), and defines the policy as a softmax over these preferences: \[\pi(a|s, \theta) = \frac{\exp(h(s,a,\theta))}{\sum_{b} \exp(h(s,b,\theta))}\] The preferences are updated via gradient ascent on the expected return, \(J(\theta)\). How does this optimization process affect the relative magnitudes of the preferences for optimal versus suboptimal actions as learning progresses? What does the policy converge to in the limit?
-
(c) The Critical Difference: Based on your analysis of their convergence properties in (a) and (b), differentiate between these two approaches. Explain which method should be considered a principled policy optimization algorithm and which is better described as a behavioral heuristic. Justify your reasoning by referencing what each method is fundamentally optimizing.