Reinforcement Learning · Lecture 7

Policy Gradients and Actor Critics

Rafael Fernandes Cunha

rafaelcunha2013.github.io

Today’s topics

1.Recap on function approximation

2.The Policy Gradient Theorem

3.Monte-Carlo Policy Gradient (REINFORCE)

4.Actor-Critic Algorithms

2

1.Explain the advantages of stochastic policies

2.Understand the Policy Gradient Theorem

3.Explain the REINFORCE algorithm

4.Explain the Actor-Critic algorithms

Learning outcomes

Value function approximation

Solution for large MDPs

Estimate value function with function approximation.

\begin{align*}
     v_{\mathbf{w}}(s) \approx v_{\pi}(s) 
\end{align*}
\begin{align*}
    q_{\mathbf{w}}(s) \approx q_{\pi}(s,a) 
\end{align*}

Update parameter w

Using MC or TD learning.

Generalize to unseen states

Monte Carlo:

TD(0):

TD():

\begin{align*}
G_t
\end{align*}
\begin{align*}
R_{t+1} + \gamma v_{\mathbf{w}}(S_{t+1})
\end{align*}
\begin{align*}
G^{\lambda}_t = R_{t+1} + \gamma \Big( (1-\lambda) v_{\mathbf{w}}(S_{t+1}) + \lambda G^{\lambda}_{t+1} \Big)
\end{align*}

Approximate Values By Stochastic Gradient Descent

\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Goal: Learn the state-value function

\begin{align*}
\mathbf{w} \leftarrow \mathbf{w} + \Delta \mathbf{w}
\end{align*}
\begin{align*}
\Delta\mathbf{w}= \alpha (v_{\pi}(S) - v_{\mathbf{w}}(S)) \nabla_{\mathbf{w}}v_{\mathbf{w}}(S)
\end{align*}

Stochastic Gradient Descent

How do we approximate the true
(target) value when sampling?

\begin{align*}
\Delta \mathbf{w} = - \frac{1}{2} \alpha \nabla_{\mathbf{w}}J(\mathbf{w})
\end{align*}

Bootstrapping methods include
only a part of the gradient. We call
them
semi-gradient methods.

Linear Function Approximation

\begin{align*}
    \mathbf{x}(s) = \begin{pmatrix}
        x_1(s) \\
        \vdots \\
        x_n(s)
    \end{pmatrix}
\end{align*}

Represent state by a feature vector

Feature construction for Linear Methods

​

➢Polynomials

​

➢Fourier Basis

​

➢Coarse Coding

​

➢Tile Coding

​

➢Radial Basis Function

(Generalization of coarse coding to continuous-valued features)

Linear Value Function Approximation for Prediction

Monte-Carlo: is an unbiased sample of true value

\begin{align*}
\Delta\mathbf{w} = \alpha (v_{\pi}(S_t) - v_{\mathbf{w}}(S_t)) \mathbf{x}_t
\end{align*}
\begin{align*}
= \alpha (G_t - v_{\mathbf{w}}(S_t)) \mathbf{x}_t
\end{align*}

TD(0): is a biased sample of true value

\begin{align*}
\Delta\mathbf{w} = \alpha (v_{\pi}(S_t) - v_{\mathbf{w}}(S_t)) \mathbf{x}_t
\end{align*}
\begin{align*}
= \alpha (R_{t+1} + \gamma v_{\mathbf{w}}(S_{t+1}) - v_{\mathbf{w}}(S_t)) \mathbf{x}_t
\end{align*}
\begin{align*}
= \alpha \delta_t \mathbf{x}_t
\end{align*}

TD-error

Action-Value Function Approximation

SARSA is TD applied to state-action pairs

\begin{align*}
  \mathbf{w}_{t+1} \doteq \mathbf{w}_t + \alpha \Big[{\color{blue}R_{t+1} + \gamma q_{\mathbf{w}_t}(S_{t+1}, A_{t+1})}- q_{\mathbf{w}}(S_t, A_t)\Big] \nabla q_{\mathbf{w}_t}(S_t, A_t)
\end{align*}

Episodic semi-gradient one-step SARSA

Convergence of Prediction Algorithms

On/Off-Policy

Algorithm

Table Lookup

Linear

Non-Linear

On-Policy

MC

TD

Off-Policy

MC

TD

Deadly triad

Bootstrapping

Off-policy

Function Approximation

General overview

Model-free RL

Monte Carlo

SARSA

q-learning

double q-learning

Expected SARSA

REINFORCE

Actor-Critics

Value-based methods

Policy-based methods

Today’s Lecture

General overview

Model-free RL

Value-based methods

Policy-based methods

Sometimes learning a policy is easier (e.g., “always move forward” is optimal)

Estimate or

Learn a policy

Usually deterministic

Learn a policy

Usually stochastic

Uses some objective function

Learns the policy directly!

Advantages x Disadvantages of Policy-based RL

➢True objective

➢Easy extended to high-dimensional or continuous action spaces

➢Can learn stochastic policies

Why do value-based methods
have difficulties to handle this?

\begin{align*}
Q_{\pi}(s,a_1) \\ Q_{\pi}(s,a_2) \\[1ex] \dots \\[1ex] Q_{\pi}(s,a_n)
\end{align*}

How to take the max of
infinite options of actions?

How would a deterministic policy look like?

Advantages

Advantages x Disadvantages of Policy-based RL

➢True objective

➢Easy extended to high-dimensional or continuous action spaces

➢Can learn stochastic policies

➢Sometimes policies are simple while values and models are complex

Advantages

Disadvantages

➢Could get stuck in local optima

➢Obtained knowledge can be specific, does not always generalize well

➢Does not necessarily extract all useful information from the data (when used in isolation)

Benefits of stochastic policies

MDP’s

Fully observable problem

There is always an optimal deterministic policy

The optimal policy may be stochastic

Real World

Not Fully observable problem

Observability

Search Space

Exploration

Search space is smoother for stochastic policies

It is possible to use gradients

Provides some “exploration” during learning

Why?

Example: Aliased Grid World

Suppose that you are representing the states by features that only says where are walls.

\begin{align*}
\phi(s) = (\overbrace{\underbrace{\;\;1\;\;}_{\text{up}}\;\underbrace{\;\;0\;\;}_{\text{right}}\;\underbrace{\;\;1\;\;}_{\text{down}}\;\underbrace{\;\;0\;\;}_{\text{left}}}^{\text{walls}=\text{state}})
\end{align*}

What is the optimal deterministic policy?

Example: Aliased Grid World

Either red or blue policies in grey
states will make the agent get
stuck and never reach the money.

What is the optimal deterministic policy?

Example: Aliased Grid World

What is the optimal stochastic policy?

Move randomly left or right in grey states

\begin{align*}
\pi_{\boldsymbol{\theta}}(\text{right} \mid \text{wall up and down}) &= 0.5 \\ \pi_{\boldsymbol{\theta}}(\text{left} \mid \text{wall up and down}) &= 0.5
\end{align*}

Will reach the goal state in a few steps with high probability

Directly learning the policy parameters, we can learn an optimal stochastic policy

Parameterized Policy

e.g., weights of a
neural network

Restriction: It needs to be
differentiable with respect
to its parameters

\begin{align*}
\pi(a|s) \approx \pi_{\boldsymbol{\theta}}(a|s) 
\end{align*}
\begin{align*}
\boldsymbol{\theta} \in \mathbb{R}^{d'} 
\end{align*}
\begin{align*}
  \text{Pr} \{A_t =a|S_t=s, \boldsymbol{\theta}_t = \boldsymbol{\theta}\}
\end{align*}

Discrete action space: Actions preferences with soft-max distribution.

\begin{align*} 
\pi_{\boldsymbol{\theta}}(a|s)  \doteq \frac{\exp^{h_{\boldsymbol{\theta}}(s,a)}}{\sum_b \exp{h_{\boldsymbol{\theta}}(s,b)}}
\end{align*}

Continuous action space: Normal distribution.

\begin{align*}
    \pi_{\boldsymbol{\theta}}(a|s)  \doteq \frac{1}{\sigma_{\boldsymbol{\theta}}(s) \sqrt{2 \pi}} \exp \Big( - \frac{(a - \mu_{\boldsymbol{\theta}}(s))^2}{2 \sigma_{\boldsymbol{\theta}}(s)^2} \Big)
\end{align*}

Parameterized function approximators.

\begin{align*}
\mu : \mathcal{S} \times \mathbb{R}^{d'} \rightarrow \mathbb{R}
\end{align*}
\begin{align*}
\sigma : \mathcal{S} \times \mathbb{R}^{d'} \rightarrow \mathbb{R}^+
\end{align*}

Parameterized Policy

Learning the policy parameter based on the gradient of some scalar
performance measure
with respect to the policy parameter.

e.g., weights of a
neural network

Restriction: It needs to be
differentiable with respect
to its parameters

\begin{align*}
\boldsymbol{\theta} \in \mathbb{R}^{d'} 
\end{align*}
\begin{align*}
  \text{Pr} \{A_t =a|S_t=s, \boldsymbol{\theta}_t = \boldsymbol{\theta}\}
\end{align*}
\begin{align*}
 \widehat{\nabla_{\theta} J(\boldsymbol{\theta}_t)} \in \mathbb{R}' 
\end{align*}

Stochastic estimate whose expectation
approximates the gradient.

\begin{align*}
\pi(a|s) \approx \pi_{\boldsymbol{\theta}}(a|s) 
\end{align*}

How to define the performance measure ?

\begin{align*}
J(\boldsymbol{\theta})\end{align*}

How can we learn ?

\begin{align*}
\boldsymbol{\theta}
\end{align*}
\begin{align*}
\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_{t} + \alpha \widehat{\nabla_{\theta} J(\boldsymbol{\theta}_t)}
\end{align*}

Stochastic gradient ascent in

\begin{align*}
J(\boldsymbol{\theta})
\end{align*}

Performance measure: Episodic return

Goal of RL agent

Find a policy that maximizes the expected return

\begin{align*}
G_t \doteq \;\sum_{k=0}^{T-t-1}\; \gamma^k R_{t+k+1}
\end{align*}
\begin{align*}
\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_{t} + \alpha \widehat{\nabla_{\theta} J(\boldsymbol{\theta}_t)}
\end{align*}

Episodic-return objective

\begin{align*}
J(\boldsymbol{\theta}) = \mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[G(\tau)]
\end{align*}

Trajectory

\begin{align*}
\tau = s_0, a_0, r_1, s_1, a_1, r_2, \ldots, s_{T-1}, a_{T-1}, r_{T}, s_T
\end{align*}

How to compute the probability of a trajectory?

\begin{align*}
p_{\boldsymbol{\theta}}(\tau)
\end{align*}

On which functions does it depend?

\begin{align*}
\pi_{\boldsymbol{\theta}}, \quad p(s'|s,a), \quad r(s,a)
\end{align*}

Return of a trajectory

How to define the performance measure ?

\begin{align*}
J(\boldsymbol{\theta})\end{align*}

Policy-Gradient derivation

Episodic-return objective

\begin{align*}
J(\boldsymbol{\theta}) = \mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[G(\tau)]
\end{align*}
\begin{align*}
=p(s_0) \prod_{t=0}^{T-1}\pi_{\theta}(a_t|s_t)p(s_{t+1}|s_t, a_t)
\end{align*}
\begin{align*}
    p_{\theta}(\tau)
\end{align*}
\begin{align*}
  \text{If }  J(\boldsymbol{\theta}) \geq J(\boldsymbol{\theta}' )  \text{ then } \pi_{\boldsymbol{\theta}}(a|s) \geq \pi_{\boldsymbol{\theta}'}(a|s) 
\end{align*}

This objective induces a natural ordering over policies:

\begin{align*}
\boldsymbol{\theta}^* = \underset{\boldsymbol{\theta}}{\arg \max}\mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[G(\tau)]
\end{align*}

Goal: Find

Policy-Gradient derivation

Episodic-return objective

\begin{align*}
J(\boldsymbol{\theta}) = \mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[G(\tau)]
\end{align*}
\begin{align*}
=p(s_0) \prod_{t=0}^{T-1}\pi_{\theta}(a_t|s_t)p(s_{t+1}|s_t, a_t)
\end{align*}
\begin{align*}
    p_{\theta}(\tau)
\end{align*}
\begin{align*}
\boldsymbol{\theta}^* = \underset{\boldsymbol{\theta}}{\arg \max}\mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[G(\tau)]
\end{align*}

Goal: Find

\begin{align*}
\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_{t} + \alpha \widehat{\nabla_{\theta} J(\boldsymbol{\theta}_t)}
\end{align*}

Stochastic gradient ascent:

Policy-Gradient derivation

Stochastic gradient ascent:

\begin{align*}
\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_{t} + \alpha \widehat{\nabla_{\theta} J(\boldsymbol{\theta}_t)}
\end{align*}
\begin{align*}
\nabla_{\boldsymbol{\theta}}J(\boldsymbol{\theta})
\end{align*}
\begin{align*}
=\nabla_{\boldsymbol{\theta}}\mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[G(\tau)] 
\end{align*}

What is the difficulty in computing this expression?

We cannot sample the gradient of an expectation!

Idea: Move the gradient “inside” the expectation and sample the expression!

Policy-Gradient derivation

Stochastic gradient ascent:

Definition of expectation

Linearity of the gradient

Chain rule

\begin{align*}
\nabla_\theta G(\tau) = 0
\end{align*}
\begin{align*}
\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_{t} + \alpha \widehat{\nabla_{\theta} J(\boldsymbol{\theta}_t)}
\end{align*}
\begin{align*}
\nabla_{\boldsymbol{\theta}}J(\boldsymbol{\theta})
\end{align*}
\begin{align*}
=\nabla_{\boldsymbol{\theta}}\mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[G(\tau)] 
\end{align*}
\begin{align*}
=  \nabla_{\boldsymbol{\theta}} \int d\tau G(\tau) p_{\boldsymbol{\theta}}(\tau) 
\end{align*}
\begin{align*}
=  \int d\tau G(\tau)\nabla_{\boldsymbol{\theta}}p_{\boldsymbol{\theta}}(\tau)  
\end{align*}
\begin{align*}
=   \int d\tau \nabla_{\boldsymbol{\theta}} (G(\tau) p_{\boldsymbol{\theta}}(\tau) )
\end{align*}
\begin{align*}
=   \int d\tau(  G(\tau)\nabla_{\boldsymbol{\theta}} p_{\boldsymbol{\theta}}(\tau) +  p_{\boldsymbol{\theta}}(\tau)\nabla_{\boldsymbol{\theta}}G(\tau) )
\end{align*}

Policy-Gradient derivation

Stochastic gradient ascent:

Definition of expectation

Linearity of the gradient

Chain rule

\begin{align*}
\nabla_\theta G(\tau) = 0
\end{align*}
\begin{align*}
\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_{t} + \alpha \widehat{\nabla_{\theta} J(\boldsymbol{\theta}_t)}
\end{align*}
\begin{align*}
\nabla_{\boldsymbol{\theta}}J(\boldsymbol{\theta})
\end{align*}
\begin{align*}
=\nabla_{\boldsymbol{\theta}}\mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[G(\tau)] 
\end{align*}
\begin{align*}
=  \nabla_{\boldsymbol{\theta}} \int d\tau G(\tau) p_{\boldsymbol{\theta}}(\tau) 
\end{align*}
\begin{align*}
=  \int d\tau G(\tau)\nabla_{\boldsymbol{\theta}}p_{\boldsymbol{\theta}}(\tau)  
\end{align*}
\begin{align*}
=   \int d\tau \nabla_{\boldsymbol{\theta}} (G(\tau) p_{\boldsymbol{\theta}}(\tau) )
\end{align*}
\begin{align*}
=   \int d\tau(  G(\tau)\nabla_{\boldsymbol{\theta}} p_{\boldsymbol{\theta}}(\tau) +  p_{\boldsymbol{\theta}}(\tau)\nabla_{\boldsymbol{\theta}}G(\tau) )
\end{align*}

How to write it in terms of expectation!

\begin{align*}
=\mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[ G(\tau)\nabla_{\boldsymbol{\theta}}p_{\boldsymbol{\theta}}(\tau)] 
\end{align*}
\begin{align*}
\int {\color{blue} p_{\boldsymbol{\theta}}(\tau)} d\tau G(\tau)\nabla_{\boldsymbol{\theta}}p_{\boldsymbol{\theta}}(\tau)  
\end{align*}

We are missing this term in our expression!

Policy-Gradient derivation

Stochastic gradient ascent:

Definition of expectation

Linearity of the gradient

Chain rule

\begin{align*}
\nabla_\theta G(\tau) = 0
\end{align*}
\begin{align*}
\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_{t} + \alpha \widehat{\nabla_{\theta} J(\boldsymbol{\theta}_t)}
\end{align*}
\begin{align*}
\nabla_{\boldsymbol{\theta}}J(\boldsymbol{\theta})
\end{align*}
\begin{align*}
=\nabla_{\boldsymbol{\theta}}\mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[G(\tau)] 
\end{align*}
\begin{align*}
=  \nabla_{\boldsymbol{\theta}} \int d\tau G(\tau) p_{\boldsymbol{\theta}}(\tau) 
\end{align*}
\begin{align*}
=  \int d\tau G(\tau)\nabla_{\boldsymbol{\theta}}p_{\boldsymbol{\theta}}(\tau)  
\end{align*}
\begin{align*}
=   \int d\tau \nabla_{\boldsymbol{\theta}} (G(\tau) p_{\boldsymbol{\theta}}(\tau) )
\end{align*}
\begin{align*}
=   \int d\tau(  G(\tau)\nabla_{\boldsymbol{\theta}} p_{\boldsymbol{\theta}}(\tau) +  p_{\boldsymbol{\theta}}(\tau)\nabla_{\boldsymbol{\theta}}G(\tau) )
\end{align*}

Derivative of a logarithm

Definition of expectation

\begin{align*}
=  \int d\tau G(\tau)p_{\boldsymbol{\theta}}(\tau) \frac{\nabla_{\boldsymbol{\theta}}p_{\boldsymbol{\theta}}(\tau)}{p_{\boldsymbol{\theta}}(\tau)}  
\end{align*}
\begin{align*}
=  \int d\tau G(\tau)p_{\boldsymbol{\theta}}(\tau) \nabla_{\boldsymbol{\theta}} \log(p_{\boldsymbol{\theta}}(\tau)) 
\end{align*}
\begin{align*}
=  \mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[ G(\tau) \nabla_{\boldsymbol{\theta}} \log(p_{\boldsymbol{\theta}}(\tau))] 
\end{align*}

Policy-Gradient derivation

Stochastic gradient ascent:

\begin{align*}
\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_{t} + \alpha \widehat{\nabla_{\theta} J(\boldsymbol{\theta}_t)}
\end{align*}
\begin{align*}
\nabla_{\boldsymbol{\theta}}J(\boldsymbol{\theta})
\end{align*}
\begin{align*}
=p(s_0) \prod_{t=0}^{T-1}\pi_{\theta}(a_t|s_t)p(s_{t+1}|s_t, a_t)
\end{align*}
\begin{align*}
    p_{\theta}(\tau)
\end{align*}
\begin{align*}
=  \mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[ G(\tau) \nabla_{\boldsymbol{\theta}} \log(p_{\boldsymbol{\theta}}(\tau))] 
\end{align*}

Now we can sample this expression!

This product can be converted into a summation by applying a logarithm!

But how to compute this gradient?

\begin{align*}
 \nabla_{\boldsymbol{\theta}} \log(p_{\boldsymbol{\theta}}(\tau)) =
\end{align*}

?

Policy-Gradient derivation

\begin{align*}
=p(s_0) \prod_{t=0}^{T-1}\pi_{\theta}(a_t|s_t)p(s_{t+1}|s_t, a_t)
\end{align*}
\begin{align*}
    p_{\theta}(\tau)
\end{align*}
\begin{align*}
 \nabla_{\boldsymbol{\theta}} \log(p_{\boldsymbol{\theta}}(\tau)) =
\end{align*}
\begin{align*}
\nabla_{\boldsymbol{\theta}} \log \Big( p(s_0) \prod_{t=0}^{T-1}\pi_{\theta}(a_t|s_t)p(s_{t+1}|s_t, a_t)\Big)
\end{align*}
\begin{align*}
= \nabla_{\boldsymbol{\theta}}\Big( \log  p(s_0)+ \sum_{t=0}^{T-1}\Big(\log \pi_{\boldsymbol{\theta}}(a_t|s_t) + \log p(s_{t+1}|s_t, a_t)\Big)\Big)
\end{align*}
\begin{align*}
= \nabla_{\boldsymbol{\theta}} \log  p(s_0)+ \nabla_{\boldsymbol{\theta}}\sum_{t=0}^{T-1}\Big(\log \pi_{\boldsymbol{\theta}}(a_t|s_t) + \log p(s_{t+1}|s_t, a_t)\Big)
\end{align*}
\begin{align*}
= \nabla_{\boldsymbol{\theta}} \log  p(s_0)+ \nabla_{\boldsymbol{\theta}}\sum_{t=0}^{T-1}\log \pi_{\boldsymbol{\theta}}(a_t|s_t) + \nabla_{\boldsymbol{\theta}}\sum_{t=0}^{T-1}\log p(s_{t+1}|s_t, a_t)
\end{align*}
\begin{align*}
=  \nabla_{\boldsymbol{\theta}}\sum_{t=0}^{T-1}\log \pi_{\boldsymbol{\theta}}(a_t|s_t) 
\end{align*}

It does not depend on theta!

\begin{align*}
\nabla_{\boldsymbol{\theta}}J(\boldsymbol{\theta})
\end{align*}
\begin{align*}
=  \mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[ G(\tau) \nabla_{\boldsymbol{\theta}} \log(p_{\boldsymbol{\theta}}(\tau))] 
\end{align*}

Policy-Gradient derivation

\begin{align*}
=p(s_0) \prod_{t=0}^{T-1}\pi_{\theta}(a_t|s_t)p(s_{t+1}|s_t, a_t)
\end{align*}
\begin{align*}
    p_{\theta}(\tau)
\end{align*}
\begin{align*}
 \nabla_{\boldsymbol{\theta}} \log(p_{\boldsymbol{\theta}}(\tau)) =
\end{align*}
\begin{align*}
=  \nabla_{\boldsymbol{\theta}}\sum_{t=0}^{T-1}\log \pi_{\boldsymbol{\theta}}(a_t|s_t) 
\end{align*}

This probability depends on the policy
and the transition function.

This term only depends on the policy!

\begin{align*}
\nabla_{\boldsymbol{\theta}}J(\boldsymbol{\theta})
\end{align*}
\begin{align*}
=  \mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[ G(\tau) \nabla_{\boldsymbol{\theta}} \log(p_{\boldsymbol{\theta}}(\tau))] 
\end{align*}

Policy-Gradient derivation

\begin{align*}
=p(s_0) \prod_{t=0}^{T-1}\pi_{\theta}(a_t|s_t)p(s_{t+1}|s_t, a_t)
\end{align*}
\begin{align*}
    p_{\theta}(\tau)
\end{align*}
\begin{align*}
 \nabla_{\boldsymbol{\theta}} \log(p_{\boldsymbol{\theta}}(\tau)) =
\end{align*}
\begin{align*}
=  \nabla_{\boldsymbol{\theta}}\sum_{t=0}^{T-1}\log \pi_{\boldsymbol{\theta}}(a_t|s_t) 
\end{align*}
\begin{align*}
\nabla_{\boldsymbol{\theta}}J(\boldsymbol{\theta})
\end{align*}
\begin{align*}
=  \mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[ G(\tau) \nabla_{\boldsymbol{\theta}} \log(p_{\boldsymbol{\theta}}(\tau))] 
\end{align*}
\begin{align*}
=  \mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[ G(\tau)  \nabla_{\boldsymbol{\theta}}\sum_{t=0}^{T-1}\log \pi_{\boldsymbol{\theta}}(a_t|s_t) ] 
\end{align*}
\begin{align*}
=  \mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}\Big[ \Big( \sum_{t=0}^{T-1} \nabla_{\boldsymbol{\theta}}\log \pi_{\boldsymbol{\theta}}(a_t|s_t)\Big) G(\tau)\Big] 
\end{align*}
\begin{align*}
=  \mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}\Big[ \Big( \sum_{t=0}^{T-1} \nabla_{\boldsymbol{\theta}}\log \pi_{\boldsymbol{\theta}}(a_t|s_t)\Big)\Big( \sum_{t=0}^{T-1}r(s_t, a_t) \Big)\Big] 
\end{align*}

Let’s sample this expression!

\begin{align*}
\approx  \frac{1}{N} \sum_{i=1}^N \Big( \sum_{t=0}^{T-1} \nabla_{\boldsymbol{\theta}}\log \pi_{\boldsymbol{\theta}}(a_t|s_t)\Big)\Big( \sum_{t=0}^{T-1}r(s_{i,t}, a_{i,t}) \Big) 
\end{align*}

Policy-Gradient Theorem

\begin{align*}
\nabla_{\boldsymbol{\theta}}J(\boldsymbol{\theta})
\end{align*}
\begin{align*}
\approx  \frac{1}{N} \sum_{i=1}^N \Big( \sum_{t=0}^{T-1} \nabla_{\boldsymbol{\theta}}\log \pi_{\boldsymbol{\theta}}(a_t|s_t)\Big)\Big( \sum_{t=0}^{T-1}r(s_{i,t}, a_{i,t}) \Big) 
\end{align*}
\begin{align*}
\boldsymbol{\theta}^* = \underset{\boldsymbol{\theta}}{\arg \max}\mathbb{E}_{\tau \sim p_{\boldsymbol{\theta}}(\tau)}[G(\tau)]
\end{align*}

Goal: Find

\begin{align*}
\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_{t} + \alpha \widehat{\nabla_{\theta} J(\boldsymbol{\theta}_t)}
\end{align*}

The update is

directly proportional to the return of the trajectory

inversely proportional to the policy

\begin{align*}
\nabla_\theta \log \pi(a_t|s_t;\theta) = \frac{\nabla_\theta \pi(a_t|s_t;\theta)}{\pi(a_t|s_t;\theta)}
\end{align*}

The gradient

rises the log probabilities of the good trajectories

lowers the log probabilities of the bad trajectories

REINFORCE

Variance problem on REINFORCE

The REINFORCE algorithm is based on the return of an episode!

High variance in the return

Actions have randomness because they are
sampled from a probability distribution.

The starting state may vary per episode.

The environment transition function may be stochastic.

Reducing the variance: Causality

\begin{align*}
\approx  \frac{1}{N} \sum_{i=1}^N  \sum_{t=0}^{T-1} \nabla_{\boldsymbol{\theta}}\log \pi_{\boldsymbol{\theta}}(a_t|s_t)\Big( \sum_{t'={\color{red}t}}^{T-1}r(s_{i,t'}, a_{i,t'}) \Big) 
\end{align*}
\begin{align*}
\nabla_{\boldsymbol{\theta}}J(\boldsymbol{\theta})
\end{align*}
\begin{align*}
\approx  \frac{1}{N} \sum_{i=1}^N \Big( \sum_{t=0}^{T-1} \nabla_{\boldsymbol{\theta}}\log \pi_{\boldsymbol{\theta}}(a_t|s_t)\Big)\Big( \sum_{t=0}^{T-1}r(s_{i,t}, a_{i,t}) \Big) 
\end{align*}

Causality: policy at time cannot affect reward at time when

Reducing the variance: Causality

The action-value function!

\begin{align*}
\hat{Q}_{i,t} \approx Q(s,a)
\end{align*}
\begin{align*}
\approx  \frac{1}{N} \sum_{i=1}^N  \sum_{t=0}^{T-1} \nabla_{\boldsymbol{\theta}}\log \pi_{\boldsymbol{\theta}}(a_t|s_t)\Big( \sum_{t'={\color{red}t}}^{T-1}r(s_{i,t'}, a_{i,t'}) \Big) 
\end{align*}
\begin{align*}
\nabla_{\boldsymbol{\theta}}J(\boldsymbol{\theta})
\end{align*}
\begin{align*}
\approx  \frac{1}{N} \sum_{i=1}^N \Big( \sum_{t=0}^{T-1} \nabla_{\boldsymbol{\theta}}\log \pi_{\boldsymbol{\theta}}(a_t|s_t)\Big)\Big( \sum_{t=0}^{T-1}r(s_{i,t}, a_{i,t}) \Big) 
\end{align*}

Causality: policy at time cannot affect reward at time when

reward “to-go”

\begin{align*}
\nabla_{\boldsymbol{\theta}}J(\boldsymbol{\theta})
\end{align*}
\begin{align*}
\approx  \frac{1}{N} \sum_{i=1}^N  \sum_{t=0}^{T-1} \nabla_{\boldsymbol{\theta}}\log \pi_{\boldsymbol{\theta}}(a_t|s_t)\hat{Q}_{i,t} 
\end{align*}

Which function has a similar definition?

Reducing the variance: Baseline

Without Baseline

With Baseline

All positive

Subtracting a baseline independent from the action.

\begin{align*}
\nabla_\theta J(\theta) = \mathbb{E}_{\tau\sim p_\theta(\tau)}\Big[\nabla_\theta \log p_\theta(\tau)(G_t(\tau) - b)\Big]
\end{align*}

Example: Average return (not the best baseline)

\begin{align*}
b = \frac{1}{N}\sum_{i=1}^{N} G(\tau_i)
\end{align*}

Why the baseline does not introduce bias in the gradient?

Reducing the variance: Baseline

\begin{align*}
\mathbb{E}_{\tau \sim p_\theta(\tau)} \Big[ \nabla_\theta \log p_\theta(\tau) b \Big] &= \int p_\theta(\tau) \nabla_\theta \log p_\theta(\tau) b d\tau \\ &= \int p_\theta(\tau) \frac{\nabla_\theta p_\theta(\tau)}{p_\theta(\tau)} b d\tau \\ &= \int \nabla_\theta p_\theta(\tau) b d\tau \\ &= b \nabla_\theta \int p_\theta(\tau) d\tau \\ &= b \nabla_\theta 1 \\ &= 0
\end{align*}

Subtracting a baseline is unbiased in expectation

\begin{align*}
\mathbb{E}_{\tau \sim p_\theta(\tau)} \Big[ \nabla_\theta \log p_\theta(\tau) b \Big] &= \int p_\theta(\tau) \nabla_\theta \log p_\theta(\tau) b d\tau \\ &= \int p_\theta(\tau) \frac{\nabla_\theta p_\theta(\tau)}{p_\theta(\tau)} b d\tau \\ &= \int \nabla_\theta p_\theta(\tau) b d\tau \\ &= b \nabla_\theta \int p_\theta(\tau) d\tau \\ &= b \nabla_\theta 1 \\ &= 0
\end{align*}
\begin{align*}
\mathbb{E}_{\tau \sim p_\theta(\tau)} \Big[ \nabla_\theta \log p_\theta(\tau) b \Big] &= \int p_\theta(\tau) \nabla_\theta \log p_\theta(\tau) b d\tau \\ &= \int p_\theta(\tau) \frac{\nabla_\theta p_\theta(\tau)}{p_\theta(\tau)} b d\tau \\ &= \int \nabla_\theta p_\theta(\tau) b d\tau \\ &= b \nabla_\theta \int p_\theta(\tau) d\tau \\ &= b \nabla_\theta 1 \\ &= 0
\end{align*}
\begin{align*}
\mathbb{E}_{\tau \sim p_\theta(\tau)} \Big[ \nabla_\theta \log p_\theta(\tau) b \Big] &= \int p_\theta(\tau) \nabla_\theta \log p_\theta(\tau) b d\tau \\ &= \int p_\theta(\tau) \frac{\nabla_\theta p_\theta(\tau)}{p_\theta(\tau)} b d\tau \\ &= \int \nabla_\theta p_\theta(\tau) b d\tau \\ &= b \nabla_\theta \int p_\theta(\tau) d\tau \\ &= b \nabla_\theta 1 \\ &= 0
\end{align*}
\begin{align*}
\mathbb{E}_{\tau \sim p_\theta(\tau)} \Big[ \nabla_\theta \log p_\theta(\tau) b \Big] &= \int p_\theta(\tau) \nabla_\theta \log p_\theta(\tau) b d\tau \\ &= \int p_\theta(\tau) \frac{\nabla_\theta p_\theta(\tau)}{p_\theta(\tau)} b d\tau \\ &= \int \nabla_\theta p_\theta(\tau) b d\tau \\ &= b \nabla_\theta \int p_\theta(\tau) d\tau \\ &= b \nabla_\theta 1 \\ &= 0
\end{align*}
\begin{align*}
\mathbb{E}_{\tau \sim p_\theta(\tau)} \Big[ \nabla_\theta \log p_\theta(\tau) b \Big] &= \int p_\theta(\tau) \nabla_\theta \log p_\theta(\tau) b d\tau \\ &= \int p_\theta(\tau) \frac{\nabla_\theta p_\theta(\tau)}{p_\theta(\tau)} b d\tau \\ &= \int \nabla_\theta p_\theta(\tau) b d\tau \\ &= b \nabla_\theta \int p_\theta(\tau) d\tau \\ &= b \nabla_\theta 1 \\ &= 0
\end{align*}

REINFORCE with baseline

\begin{align*}
\nabla_\theta J(\pi_\theta) &= \mathbb{E}_{\tau\sim\pi_\theta}\left[\,\sum_{t=0}^{T} G_t(\tau)\nabla_\theta \log \pi_\theta(a_t|s_t)\right] \\ &\approx \mathbb{E}_{\tau\sim\pi_\theta}\left[\,\sum_{t=0}^{T}(\,G_t(\tau) - b(s))\nabla_\theta \log \pi_\theta(a_t|s_t)\right]
\end{align*}
\begin{align*}
\nabla_\theta J(\pi_\theta) &= \mathbb{E}_{\tau\sim\pi_\theta}\left[\,\sum_{t=0}^{T} G_t(\tau)\nabla_\theta \log \pi_\theta(a_t|s_t)\right] \\ &\approx \mathbb{E}_{\tau\sim\pi_\theta}\left[\,\sum_{t=0}^{T}(\,G_t(\tau) - b(s))\nabla_\theta \log \pi_\theta(a_t|s_t)\right]
\end{align*}

If is , this choice of baselines motivates the Actor-Critic family of algorithms.

Value function parameterized by

Learn the value function via:

MC -> REINFORCE with baseline

TD-learning -> Actor-Critic

REINFORCE with baseline

Baseline

REINFORCE with baseline

Two learning rates

Actor-Critic algorithms

6

A policy that controls
how our agent acts.

A value function that
measures how good
the action taken is.

Value-Based

Policy-Based

DQN

REINFORCE

Actor-Critic

A2C

Critic

Actor

Actor-Critic algorithms

6

A policy that controls
how our agent acts.

A value function that
measures how good
the action taken is.

Learn two function approximations!

Value-Based

Policy-Based

DQN

REINFORCE

Actor-Critic

A2C

Critic

Actor

Actor-Critic algorithms

6

A policy that controls
how our agent acts.

A value function that
measures how good
the action taken is.

Value-Based

Policy-Based

DQN

REINFORCE

Actor-Critic

A2C

Critic

Actor

Actor-Critic algorithms

6

A policy that controls
how our agent acts.

A value function that
measures how good
the action taken is.

\begin{align*}
V^{\pi}(s) = \underset{\tau\sim\pi}{\mathrm{E}}\,\left[R(\tau)\,|s_0 = s\right]
\end{align*}
\begin{align*}
Q^{\pi}(s,a) = \underset{\tau\sim\pi}{\mathrm{E}}\,\left[R(\tau)\,|s_0 = s, a_0 = a\right]
\end{align*}

Value-Based

Policy-Based

DQN

REINFORCE

Actor-Critic

A2C

Critic

Actor

Advantage function

Case 1

\begin{align*}
Q^{\pi}(s,a) = -90
\end{align*}

Case 2

Inform the relative improvement caused by action a.

\begin{align*}
A^{\pi}(s_t,a_t) = Q^{\pi}(s_t,a_t) - V^{\pi}(s_t)
\end{align*}

Advantage function!

\begin{align*}
A^{\pi}(s,a) = 10
\end{align*}
\begin{align*}
A^{\pi}(s,a) = 10
\end{align*}

Same Value!

7

CRITIC

\begin{align*}
G_t(\tau)\nabla_\theta \log \pi_\theta(a_t|s_t)
\end{align*}

Policy gradient step

Direction of the steepest increase
of the (log) probability of
selecting action
from state

Return

Learns parametrized policy .

8

REINFORCE

\begin{align*}
G_t(\tau)\nabla_\theta \log \pi_\theta(a_t|s_t)
\end{align*}

Policy gradient step

Return

8

REINFORCE

The entire episode must
finish. High variance.

Learns parametrized policy .

\begin{align*}
G_t(\tau)\nabla_\theta \log \pi_\theta(a_t|s_t)
\end{align*}

Policy gradient step

Return

8

REINFORCE

Can we use something better?

Learns parametrized policy .

\begin{align*}
G_t(\tau)\nabla_\theta \log \pi_\theta(a_t|s_t)
\end{align*}

Policy gradient step

Return

8

REINFORCE

Estimate the relative improvement of
action
at state . Low variance.

\begin{align*}
\textit{Actor-Critic:}\quad \nabla_\theta J(\pi_\theta) &= \mathbb{E}_t\big[A_t^{\pi}\nabla_\theta \log \pi_\theta(a_t \mid s_t)\big] \\ \textit{REINFORCE:}\quad \nabla_\theta J(\pi_\theta) &= \mathbb{E}_t\big[R_t(\tau)\nabla_\theta \log \pi_\theta(a_t \mid s_t)\big]
\end{align*}

A2C

Advantage

Learns parametrized policy .

Block Diagram of A2C algorithm

\begin{align*}
A^{\pi}(s_t,a_t) = Q^{\pi}(s_t,a_t) - V^{\pi}(s_t)
\end{align*}
\begin{align*}
\delta_t = r_t + \gamma V^{\pi}(s_{t+1}) - V^{\pi}(s_t)
\end{align*}

Time Difference (TD) error

Recall the advantage function

Mnih, Volodymyr, et al. "Asynchronous methods for deep reinforcement learning." International conference on machine learning. PMLR, 2016.

9

Agent

TD-error calculation

Update Critic

Update Actor

REINFORCE with baseline

Policy gradient algorithms (REINFORCE)

•Suitable for continuous state and action space.

•High variance.

Value based algorithms

•Not suitable for continuous action space.

Summary on policy gradient algorithms

Actor-Critic algorithm

•Suitable for continuous state and action space.

•Low variance.

10

Taxonomy of Deep

RL algorithms

Take home message

•Policy gradient algorithms learn the policy directly

•It is possible to learn stochastic policies

•Suitable for continuous action space problems

•Choosing as baseline introduce actor-critic algorithms

•Actor-critic algorithms use policy gradients and value functions

•Actor-critic algorithms reduce the variance problems of REINFORCE

​

Take home message