Exercise 1.3.3

Weighted-average method: Q-values at t = 9 from a reward table, α = 0.5

Reinforcement Learning › Exercises › Lecture 1 · Exploration Methods › 1.3 Weighted-average method (nonstationary problems)
EasyWritten

Consider a \(k\)-armed bandit problem with \(k=4\) actions, \(a \in \{1, 2, 3, 4\}\). Consider initial estimates of \(Q_1(a) = 0\), for all \(a\). Table 1 contains the observed rewards \(R_t\) for the actions \(A_t = k\) taken at each time step t.

Table 1: Sequence of rewards received in a \(4\)-armed bandit problem.
Time \(t\) Action \(1\) Action \(2\) Action \(3\) Action \(4\)
\(t=1\) \(-3\)
\(t=2\) \(4\)
\(t=3\) \(2\)
\(t=4\) \(9\)
\(t=5\) \(4.5\)
\(t=6\) \(1\)
\(t=7\) \(-2.5\)
\(t=8\) \(5\)

Calculate the estimated \(Q\)-value (\(Q_t(a)\)) for time step \(t=9\) for each action using the weighted-average method with \(\alpha = 0.5\).