Exercise 1.9.3
Action-value methods and policies: probabilities of each action at t = 9
Consider a \(k\)-armed bandit problem with \(k=4\) actions, \(a \in \{1, 2, 3, 4\}\). Consider initial estimates of \(Q_1(a) = 0\), for all \(a\). Table 1 contains the observed rewards \(R_t\) for the actions \(A_t = k\) taken at each time step t.
| Time \(t\) | Action \(1\) | Action \(2\) | Action \(3\) | Action \(4\) |
|---|---|---|---|---|
| \(t=1\) | \(-3\) | |||
| \(t=2\) | \(4\) | |||
| \(t=3\) | \(2\) | |||
| \(t=4\) | \(9\) | |||
| \(t=5\) | \(4.5\) | |||
| \(t=6\) | \(1\) | |||
| \(t=7\) | \(-2.5\) | |||
| \(t=8\) | \(5\) |
Recall that for action-value methods, you must first select the method you will use to update the action-value function and then apply the selected policy using the estimation of the action-value function obtained.
Fill in Table 3 with the probability of selecting each action at time \(t=9\), rounded to three decimal places, using the methods listed below. Provide the equations that you used. If you decide to write Python code to answer this question, please attach it to your answer. For the action-value methods, consider the estimated action-values in Table 2, and the number of times each action was selected as shown in Table 1. For the action preference method, consider Table 1 and \(H_1(a) = 0\) for all \(a\).
-
a) Sample-Average method and:
1. Greedy policy.
2. \(\epsilon\)-greedy policy with \(\epsilon=0.2\).
3. Upper confidence bound policy with \(c=2\).
4. Softmax policy with \(\tau=2\).
-
b) Weight-Average method with \(\alpha = 0.5\) and:
1. Greedy policy.
2. \(\epsilon\)-greedy policy with \(\epsilon=0.2\).
3. Upper confidence bound policy with \(c=2\).
4. Softmax policy with \(\tau=2\).
c) Action Preference (\(\alpha=0.25\)).
d) Action Preference with baseline (\(\alpha=0.25\)).
| Method | \(Q_9(1)\) | \(Q_9(2)\) | \(Q_9(3)\) | \(Q_9(4)\) |
|---|---|---|---|---|
| Sample-Average | \(0.8\) | \(3.8\) | \(3.0\) | \(3.2\) |
| Weight-Average with \(\alpha=0.5\) | \(1.4\) | \(2.0\) | \(2.8\) | \(1.5\) |