Exercise 2.3.2
Returns and discounting: G_2 with γ = 0.5 and five rewards
Suppose a discount factor \(\gamma = 0.5\) and we observe the following sequence of rewards: \(R_1 = 2, R_2 = 1.75, R_3 = -2, R_4 = 3\), and \(R_5 = 4\), with \(T = 5\), where \(T\) is the time-step representing the terminal state. What is \(G_2\), where \(G_t\) is the discounted return at time \(t\)? Recall that the discounted return \(G_t\) at time \(t\) is defined as the sum of discounted rewards received after time \(t\), mathematically represented as: \[\begin{equation*} G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots + \gamma^{T-1} R_{T} \end{equation*}\] where \(R_{t+k}\) is the reward received at time \(t+k\), \(k \in \{1, \ldots, T-t\}\), and \(\gamma \in [0,1]\) is the discount factor.
Adapted from Sutton & Barto (2018), Reinforcement Learning: An Introduction, 2nd ed., Exercise 3.8. Licensed CC BY-NC-ND 2.0.