Exercise 2.6.1
Bellman equation in matrix form: r^π and P^π, equiprobable policy
Consider the MDP given in Figure 1. Actions have a \(0.9\) probability of success and with \(0.1\) probability we remain in the same state. \(R_t =0\) for all transitions that end up in \(S_0\), and \(R_t = -1\) for all other transitions. Consider a discount factor of \(\gamma = 0.9\).
Recall the Bellman equation \[\begin{align*} v_{\pi}(s) = \sum_a \pi(a|s) \sum_{s' , r} p(s' , r|s,a) [r + \gamma v_{\pi}(s')] \end{align*}\] The Bellman value equation, for a given \(\pi\), can be expressed using matrices, \[\begin{align*} \mathbf{v} = \mathbf{r}^{\pi} + \gamma \mathbf{P}^{\pi} \mathbf{v} \end{align*}\] where the elements of these matrices are defined as \[\begin{align*} v_i &\equiv v(s_i) \\ r_i^{\pi} &\equiv \mathbb{E}[R_{t+1}|S_t = s_i, A_t \sim \pi] \\ P_{ij}^{\pi} &\equiv \sum_a \pi(a|s_i)p(s_j|s_i, a) \end{align*}\] For the equiprobable random policy, provide the vector \(\mathbf{r}^{\pi}\) and the matrix \(\mathbf{P}^{\pi}\), given that \[\begin{align*} \mathbf{v} \equiv \begin{bmatrix} v(s_0) \\ v(s_1) \\ v(s_2) \end{bmatrix} \end{align*}\]