Exercise 1.3.3
Weighted-average method: Q-values at t = 9 from a reward table, α = 0.5
Reinforcement Learning › Exercises › Lecture 1 · Exploration Methods › 1.3 Weighted-average method (nonstationary problems)
Consider a \(k\)-armed bandit problem with \(k=4\) actions, \(a \in \{1, 2, 3, 4\}\). Consider initial estimates of \(Q_1(a) = 0\), for all \(a\). Table 1 contains the observed rewards \(R_t\) for the actions \(A_t = k\) taken at each time step t.
| Time \(t\) | Action \(1\) | Action \(2\) | Action \(3\) | Action \(4\) |
|---|---|---|---|---|
| \(t=1\) | \(-3\) | |||
| \(t=2\) | \(4\) | |||
| \(t=3\) | \(2\) | |||
| \(t=4\) | \(9\) | |||
| \(t=5\) | \(4.5\) | |||
| \(t=6\) | \(1\) | |||
| \(t=7\) | \(-2.5\) | |||
| \(t=8\) | \(5\) |
Calculate the estimated \(Q\)-value (\(Q_t(a)\)) for time step \(t=9\) for each action using the weighted-average method with \(\alpha = 0.5\).