Exercise 1.2.1
Sample-average method: Q-values at t = 9 from a reward table
Consider a \(k\)-armed bandit problem with \(k=4\) actions, \(a \in \{1, 2, 3, 4\}\). Consider initial estimates of \(Q_1(a) = 0\), for all \(a\). Table 1 contains the observed rewards \(R_t\) for the actions \(A_t = k\) taken at each time step t.
| Time \(t\) | Action \(1\) | Action \(2\) | Action \(3\) | Action \(4\) |
|---|---|---|---|---|
| \(t=1\) | \(-3\) | |||
| \(t=2\) | \(4\) | |||
| \(t=3\) | \(2\) | |||
| \(t=4\) | \(9\) | |||
| \(t=5\) | \(4.5\) | |||
| \(t=6\) | \(1\) | |||
| \(t=7\) | \(-2.5\) | |||
| \(t=8\) | \(5\) |
Calculate the estimated \(Q\)-value (\(Q_t(a)\)) for time step \(t=9\) for each action using the sample-average method.