Exercise R.3.1

Expected value: return of policies in a three-cell corridor

Reinforcement Learning › Exercises › Background review › R.3 Expected value
EasyWritten

Consider the environment:

diagram

Assume the agent starts in the middle cell (\(S\)). If the agent receives a reward, the episode ends. The left cell gives a reward of 1, and the right cell of -1. Let \(L\) and \(R\) denote the only possible actions, namely left and right.

  • What is the expected return of the following policies given the starting state \(S\)?

    • \(\pi(L \mid S)=0.4\)

    • \(\pi(L \mid S)=0.2\)

    • \(\pi(R \mid S)=0.6\)

  • The expected value of the return from a policy given the starting state \(S\) is 0.5. What is the policy?

  • What is the policy that maximizes the expected return? Is it stochastic or deterministic?