Exercise 5.3.4

SARSA/Q-learning: numeric update of Q(s1,1) from a Q-table

Reinforcement Learning › Exercises › Lecture 5 · Model-Free Control › 5.3 Q-learning (off-policy learning)
EasyWritten

Given the following table of Q-values for three states (\(s_1, s_2, s_3\)) and two actions (1, 2), calculate the updated Q-value \(Q(s, a)\) for the state-action pair \(Q(s_1, 1)\) after a transition to state \(s_2\) with a reward of 5. Assume \(\alpha = 0.1\) and \(\gamma = 0.9\). Consider that the behavior policy is the greedy policy.

State Action 1 Action 2
\(s_1\) 4.0 4.5
\(s_2\) 8.0 1.0
\(s_3\) 3.5 2.0