Reinforcement Learning · Lecture 6

Function Approximation

Rafael Fernandes Cunha

rafaelcunha2013.github.io

Today’s topics

1.Recap Model-Free Control

2.Value Function approximation (VFA)

3.Classes of VFA

1.Tabular

2.State Aggregation

3.Linear

4.Non-linear

4.Stochastic Gradient Descent (SGD) with VFA

5.Linear VFA

6.Prediction and Control with VFA

2

1.Describe the different types of VFA

2.Apply the SGD algorithm to a linear VFA

3.Explain what feature vectors are

4.Explain the VFA update expression for MC, TD
and SARSA methods

5.Explain why TD is not a true SGD algorithm

Learning outcomes

Behaviour vs Target policy

Behavior policy: Generate the action the agent will actually perform in the environment.

Target policy: The policy we want to learn.

Behaviour vs Target policy

The simulation is currently at state

Which policy

➢… chooses the action that the agent will actually perform in the environment?

➢… leads the environment to produce and ?

➢… generates the action used solely for estimation?

➢… determines how to evaluate the next state , for the learning update ?

➢… defines the action-value function is used to compute ?

➢… is the policy we are trying to learn and improve?

➢… needs to ensure sufficient exploration?

➢… defines the Bellman expectation equations?

Approach to ensure all actions are selected infinitely often

On-policy: Learn about target policy from experience sampled from behaviour policy

Off-policy: Learn about target policy from experience sampled from behaviour policy

target policy = behaviour policy

target policy behaviour policy

Action-value update equations

Correct or Wrong?

✅

❌

✅

❌

SARSA vs Q-learning

greedy

-greedy

SARSA (Behavior policy = Target policy)

Q-learning

Policy must be GLIE to assure convergence.

Behaviour policy should provide infinite exploration.

\begin{align*}
    \max_{a'} q_t(S_{t+1}, a') = q_t \Big( S_{t+1}, \underset{a'}{\argmax} \; q_t(S_{t+1}, a') \Big)
\end{align*}

Q-learning overestimation and Double q-learning

Selecting the action

Evaluating the action

Q-learning overestimates because:

1) It uses the same action-value function to select and to evaluate the action!

2) It has the maximization term.

\begin{align*}
    \textcolor{blue}{q_2}(S_t, A_t) \leftarrow \textcolor{blue}{q_2}(S_t, A_t) + \alpha \big[R_{t+1} + \gamma \textcolor{red}{q_1}\Big(S_{t+1}, \argmax_a \textcolor{blue}{q_2}(S_{t+1}, a)\Big) - \textcolor{blue}{q_2}(S_t, A_t)\Big] 
\end{align*}
\begin{align*}
        \textcolor{red}{q_1}(S_t, A_t) \leftarrow \textcolor{red}{q_1}(S_t, A_t) + \alpha \big[R_{t+1} + \gamma \textcolor{blue}{q_2}\Big(S_{t+1}, \argmax_a \textcolor{red}{q_1}(S_{t+1}, a)\Big) - \textcolor{red}{q_1}(S_t, A_t)\Big] 
\end{align*}

DQN

Q-learning with tricks to help convergence

➢Replay Buffer: Sampling from this buffer helps get less correlated data.

➢Target Network: Avoid chasing a moving target

Off-Policy Learning: Importance Sampling Corrections

Off-policy: Learn about target policy from experience sampled from behaviour policy

Evaluate target policy
to compute
or

Use behaviour policy
to generate actions

​

➢Note that you are sampling from one probability distribution to learn about a different one.

​

\begin{align*}
        \mathbb{E}[R_{t+1}|S_t = s, A_t \sim \pi] 
\end{align*}
\begin{align*}
   = \mathbb{E} \Big[\frac{\pi(A_t|S_t)}{\mu(A_t|S_t)}R_{t+1} | S_t = s, A_t \sim \mu \Big] 
\end{align*}

Incremental Update rule

NewEstimate

OldEstimate

StepSize

InformativeValue

Estimate

Target

OldEstimate

-

Control:

Each algorithm changes only the Target value!

Algorithms: Model-Free Control

Monte Carlo

SARSA

Q-learning

Expected SARSA

Double Q-learning

Greedy policy

Algorithms: Model-Free Control

Monte Carlo

SARSA

Q-learning

Expected SARSA

Double Q-learning

Greedy policy

Function Approximation

Today’s lecture!

Function Approximation

​

➢The policy, value function, and model, are all functions.

➢We want to learn these from experience.

➢If there are too many states, we need to approximate.

​

We consider learning predictions (value functions; including value-based control)

​

Why function approximation?

Large-Scale Reinforcement Learning

Reinforcement learning can be used to solve large problems, e.g.

​

➢Backgammon: states

➢Go: states

➢Helicopter: continuous state space

➢Robots: real world

​

How can we apply our methods for prediction and control?

Value function approximation

So far we mostly considered lookup tables

➢Every state s has an entry v(s)

➢Or every state-action pair has an entry

​

Problem with large MDPs:

➢There are too many states and/or actions to store in memory

➢It is too slow to learn the value of each state individually

➢Individual environment states are often not fully observable

​

Value function approximation

Solution for large MDPs

Estimate value function with function approximation.

\begin{align*}
     v_{\mathbf{w}}(s) \approx v_{\pi}(s) 
\end{align*}
\begin{align*}
    q_{\mathbf{w}}(s) \approx q_{\pi}(s,a) 
\end{align*}

Update parameter w

How?

Using MC or TD learning.

Generalize to unseen states

Classes of Function Approximation

➢Tabular: a table with an entry for each MDP state

➢State aggregation: Partition environment states (or observations) into a discrete set

➢Linear function approximation

➢Consider fixed agent state update (e.g., )

➢Fixed feature map

➢Values are linear function of features:

➢Note: state aggregation and tabular are special cases of linear FA

➢Differentiable function approximation

➢ is a differentiable function of w, could be non-linear

➢E.g., a convolutional neural network that takes pixels as input

➢Another interpretation: features are not fixed, but learned

Classes of Function Approximation

In principle, any function approximator can be used, but RL has specific properties:

​

➢Experience is not i.i.d. — successive time-steps are correlated

➢Agent’s policy affects the data it receives

➢Regression targets can be non-stationary

➢...because of changing policies (which can change the target and the data!)

➢...because of bootstrapping

➢...because of non-stationary dynamics (e.g., other learning agents)

➢...because the world is large (never quite in the same state)

Classes of Function Approximation

Which function approximation should you choose?

Tabular: good theory but does not scale/generalize

Linear: reasonably good theory, but requires good features

Non-linear: less well-understood, but scales well

➢Flexible, and less reliant on picking good features first (e.g., by hand)

➢(Deep) neural nets often perform quite well, and remain a popular choice

Approximate Values By Stochastic Gradient Descent

Goal: Learn the state-value function

\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

True value

\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Estimated value
parameterized by
w

\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Approximate Values By Stochastic Gradient Descent

\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Goal: Learn the state-value function

\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

True value

Estimated value
parameterized by
w

\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}
\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Loss function

Approximate Values By Stochastic Gradient Descent

\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Goal: Learn the state-value function

Find w that minimize this difference!

Goal:

Approximate Values By Stochastic Gradient Descent

\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Goal: Learn the state-value function

Let it be a differentiable
function of parameter vector
w

\begin{align*}
   \nabla_{\mathbf{w}}J(\mathbf{w}) = \begin{pmatrix} 
    \frac{\partial J(\mathbf{w})}{\partial w_1} \\ 
    \vdots \\ 
    \frac{\partial J(\mathbf{w})}{\partial w_n} 
    \end{pmatrix}
\end{align*}

Gradient

Goal: to minimize

\begin{align*}
J(\mathbf{w})
\end{align*}
\begin{align*}
\Delta \mathbf{w} = - \frac{1}{2} \alpha \nabla_{\mathbf{w}}J(\mathbf{w})
\end{align*}

Method: move w in the direction of negative gradient

step-size parameter

Update rule:

\begin{align*}
\mathbf{w} \leftarrow \mathbf{w} + \Delta \mathbf{w}
\end{align*}

Approximate Values By Stochastic Gradient Descent

\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Goal: Learn the state-value function

\begin{align*}
\mathbf{w} \leftarrow \mathbf{w} + \Delta \mathbf{w}
\end{align*}
\begin{align*}
\Delta \mathbf{w} = - \frac{1}{2} \alpha \nabla_{\mathbf{w}}J(\mathbf{w})
\end{align*}

Approximate Values By Stochastic Gradient Descent

\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Goal: Learn the state-value function

\begin{align*}
\Delta \mathbf{w} = - \frac{1}{2} \alpha \nabla_{\mathbf{w}}J(\mathbf{w})
\end{align*}
\begin{align*}
\mathbf{w} \leftarrow \mathbf{w} + \Delta \mathbf{w}
\end{align*}

How to solve it without a model?

This is an expectation equation.

We need to sample it!

Stochastic Gradient Descent

\begin{align*}
\Delta\mathbf{w}= \alpha (v_{\pi}(S) - v_{\mathbf{w}}(S)) \nabla_{\mathbf{w}}v_{\mathbf{w}}(S)
\end{align*}
\begin{align*}
    =- \frac{1}{2} \alpha \Big(2 \mathbb{E}_{S \sim d}[v_{\pi}(S) - v_{\mathbf{w}}(S)] \nabla_{\mathbf{w}} \big(v_{\pi}(S) -v_{\mathbf{w}}(S) \big) \Big)\\
\end{align*}
\begin{align*}
   =-  \alpha  \mathbb{E}_{S \sim d}[v_{\pi}(S) - v_{\mathbf{w}}(S)] \Big(\nabla_{\mathbf{w}} v_{\pi}(S) -\nabla_{\mathbf{w}}v_{\mathbf{w}}(S) \Big) 
\end{align*}
\begin{align*}
= \alpha \mathbb{E}_{S \sim d}[v_{\pi}(S) - v_{\mathbf{w}}(S)] \nabla_{\mathbf{w}}v_{\mathbf{w}}(S)
\end{align*}

Monte Carlo:

TD(0):

TD():

\begin{align*}
G_t
\end{align*}
\begin{align*}
R_{t+1} + \gamma v_{\mathbf{w}}(S_{t+1})
\end{align*}
\begin{align*}
G^{\lambda}_t = R_{t+1} + \gamma \Big( (1-\lambda) v_{\mathbf{w}}(S_{t+1}) + \lambda G^{\lambda}_{t+1} \Big)
\end{align*}

Approximate Values By Stochastic Gradient Descent

\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Goal: Learn the state-value function

\begin{align*}
\mathbf{w} \leftarrow \mathbf{w} + \Delta \mathbf{w}
\end{align*}
\begin{align*}
\Delta\mathbf{w}= \alpha (v_{\pi}(S) - v_{\mathbf{w}}(S)) \nabla_{\mathbf{w}}v_{\mathbf{w}}(S)
\end{align*}

Stochastic Gradient Descent

How do we approximate the true
(target) value when sampling?

Monte Carlo:

TD(0):

TD():

\begin{align*}
G_t
\end{align*}
\begin{align*}
R_{t+1} + \gamma v_{\mathbf{w}}(S_{t+1})
\end{align*}
\begin{align*}
G^{\lambda}_t = R_{t+1} + \gamma \Big( (1-\lambda) v_{\mathbf{w}}(S_{t+1}) + \lambda G^{\lambda}_{t+1} \Big)
\end{align*}

Approximate Values By Stochastic Gradient Descent

\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Goal: Learn the state-value function

\begin{align*}
\mathbf{w} \leftarrow \mathbf{w} + \Delta \mathbf{w}
\end{align*}
\begin{align*}
\Delta\mathbf{w}= \alpha (v_{\pi}(S) - v_{\mathbf{w}}(S)) \nabla_{\mathbf{w}}v_{\mathbf{w}}(S)
\end{align*}

Stochastic Gradient Descent

Do you see any problem here?

\begin{align*}
\Delta \mathbf{w} = - \frac{1}{2} \alpha \nabla_{\mathbf{w}}J(\mathbf{w})
\end{align*}

Monte Carlo:

TD(0):

TD():

\begin{align*}
G_t
\end{align*}
\begin{align*}
R_{t+1} + \gamma v_{\mathbf{w}}(S_{t+1})
\end{align*}
\begin{align*}
G^{\lambda}_t = R_{t+1} + \gamma \Big( (1-\lambda) v_{\mathbf{w}}(S_{t+1}) + \lambda G^{\lambda}_{t+1} \Big)
\end{align*}

Approximate Values By Stochastic Gradient Descent

\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Goal: Learn the state-value function

\begin{align*}
\mathbf{w} \leftarrow \mathbf{w} + \Delta \mathbf{w}
\end{align*}
\begin{align*}
\Delta\mathbf{w}= \alpha (v_{\pi}(S) - v_{\mathbf{w}}(S)) \nabla_{\mathbf{w}}v_{\mathbf{w}}(S)
\end{align*}

Stochastic Gradient Descent

Do you see any problem here?

\begin{align*}
\Delta \mathbf{w} = - \frac{1}{2} \alpha \nabla_{\mathbf{w}}J(\mathbf{w})
\end{align*}

We considered that did not depend
on
w when taking the gradient!

\begin{align*}
v_{\pi}(S)
\end{align*}
\begin{align*}
\nabla_{\mathbf{w}}v_{\pi}(S) = 0
\end{align*}

TD(0):

TD():

\begin{align*}
R_{t+1} + \gamma v_{\mathbf{w}}(S_{t+1})
\end{align*}
\begin{align*}
G^{\lambda}_t = R_{t+1} + \gamma \Big( (1-\lambda) v_{\mathbf{w}}(S_{t+1}) + \lambda G^{\lambda}_{t+1} \Big)
\end{align*}

Approximate Values By Stochastic Gradient Descent

\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Goal: Learn the state-value function

\begin{align*}
\mathbf{w} \leftarrow \mathbf{w} + \Delta \mathbf{w}
\end{align*}
\begin{align*}
\Delta\mathbf{w}= \alpha (v_{\pi}(S) - v_{\mathbf{w}}(S)) \nabla_{\mathbf{w}}v_{\mathbf{w}}(S)
\end{align*}

Stochastic Gradient Descent

Do you see any problem here?

\begin{align*}
\Delta \mathbf{w} = - \frac{1}{2} \alpha \nabla_{\mathbf{w}}J(\mathbf{w})
\end{align*}

We considered that did not depend
on
w when taking the gradient!

\begin{align*}
v_{\pi}(S)
\end{align*}

Bootstrapping methods are biased! They do not produce a true gradient-descent method!

They include only a part of the gradient. We call them semi-gradient methods.

Approximate Values By Stochastic Gradient Descent

\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Goal: Learn the state-value function

\begin{align*}
\mathbf{w} \leftarrow \mathbf{w} + \Delta \mathbf{w}
\end{align*}
\begin{align*}
\Delta\mathbf{w}= \alpha (v_{\pi}(S) - v_{\mathbf{w}}(S)) \nabla_{\mathbf{w}}v_{\mathbf{w}}(S)
\end{align*}

Stochastic Gradient Descent

How to compute this gradient?

Linear

Non-linear

We need to define how we are approximating our function!

(Deep) neural nets

Example: 1000-state random walk task

Asymptotic values of semi-gradient TD are worse than the asymptotic Monte Carlo values.

Action-Value Function Approximation

SARSA is TD applied to state-action pairs

\begin{align*}
\mathbf{w}_{t+1} \doteq \mathbf{w}_t + \alpha \Big[ q_{\pi}(s,a) - q_{\mathbf{w}_t}(S_t, A_t)\Big] \nabla q_{\mathbf{w}_t}(S_t, A_t)
\end{align*}
\begin{align*}
  \mathbf{w}_{t+1} \doteq \mathbf{w}_t + \alpha \Big[{\color{blue}R_{t+1} + \gamma q_{\mathbf{w}_t}(S_{t+1}, A_{t+1})}- q_{\mathbf{w}}(S_t, A_t)\Big] \nabla q_{\mathbf{w}_t}(S_t, A_t)
\end{align*}

Episodic semi-gradient one-step SARSA

Example: Mountain Car Task (semi-gradient SARSA())

\begin{align*}
-\max_a q_{\mathbf{w}}(s,a)
\end{align*}

Cost-to-go

Linear Function Approximation

\begin{align*}
\mathbf{x} : \mathcal{S} \rightarrow \mathbb{R}^n
\end{align*}
\begin{align*}
\mathbf{x}_t = \mathbf{x}(S_t)
\end{align*}
\begin{align*}
    \mathbf{x}(s) = \begin{pmatrix}
        x_1(s) \\
        \vdots \\
        x_n(s)
    \end{pmatrix}
\end{align*}

Represent state by a feature vector

is a fixed mapping from state to features

Short-hand:

Examples:

➢Distance of robot from landmarks

➢Trends in the stock market

➢Piece and pawn configuration in chess

Linear Function Approximation

\begin{align*}
    \mathbf{x}(s) = \begin{pmatrix}
        x_1(s) \\
        \vdots \\
        x_n(s)
    \end{pmatrix}
\end{align*}

Represent state by a feature vector

Feature construction for Linear Methods

​

➢Polynomials

​

\begin{align*}
\mathbf{x}(s) = (1, s_1, s_2, s_1 s_2, s_1^2, s_2^2, s_1^2 s_2, s_1 s_2^2, s_1^2 s_2^2)
\end{align*}

Linear Function Approximation

\begin{align*}
    \mathbf{x}(s) = \begin{pmatrix}
        x_1(s) \\
        \vdots \\
        x_n(s)
    \end{pmatrix}
\end{align*}

Represent state by a feature vector

Feature construction for Linear Methods

​

➢Polynomials

​

➢Fourier Basis

​

\begin{align*}
x_i(s) = \cos(i \pi s), \quad s \in [0,1]
\end{align*}

One-dimensional Fourier cosine-basis features for approximating functions over the interval [0, 1].

Linear Function Approximation

\begin{align*}
    \mathbf{x}(s) = \begin{pmatrix}
        x_1(s) \\
        \vdots \\
        x_n(s)
    \end{pmatrix}
\end{align*}

Represent state by a feature vector

Feature construction for Linear Methods

​

➢Polynomials

​

➢Fourier Basis

​

➢Coarse Coding

​

Coarse coding provides large features vector x(s)

Parameter vector w gives a value to each feature.

➢We aggregate multiple states

➢This means the resulting feature vector/agent state is non-Markovian

➢This is the common case when using function approximation

➢Consider whether good solutions exist for given features + function approximation

➢Neural networks tend to be more flexible

Linear Function Approximation

\begin{align*}
    \mathbf{x}(s) = \begin{pmatrix}
        x_1(s) \\
        \vdots \\
        x_n(s)
    \end{pmatrix}
\end{align*}

Represent state by a feature vector

Feature construction for Linear Methods

​

➢Polynomials

​

➢Fourier Basis

​

➢Coarse Coding

​

➢Tile Coding

​

Linear Function Approximation

\begin{align*}
    \mathbf{x}(s) = \begin{pmatrix}
        x_1(s) \\
        \vdots \\
        x_n(s)
    \end{pmatrix}
\end{align*}

Represent state by a feature vector

Feature construction for Linear Methods

​

➢Polynomials

​

➢Fourier Basis

​

➢Coarse Coding

​

➢Tile Coding

​

➢Radial Basis Function

(Generalization of coarse coding to continuous-valued features)

\begin{align*}
x_i(s) \doteq \exp \Big( - \frac{||s - c_i||^2}{2 \sigma_i^2}\Big)
\end{align*}

Linear Value Function Approximation

\begin{align*}
 = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - \mathbf{w}^T\mathbf{x}(S))^2]
\end{align*}
\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Objective function:

Approximate value function by a
linear combination of features

\begin{align*}
v_{\mathbf{w}}(s) = \mathbf{w}^T\mathbf{x}(s) = \sum_{j=1}^n \mathbf{x}_j(s) \mathbf{w}_j
\end{align*}

quadratic in w

Stochastic gradient descent
converges on global optimum!

Linear Value Function Approximation

Approximate value function by a
linear combination of features

\begin{align*}
v_{\mathbf{w}}(s) = \mathbf{w}^T\mathbf{x}(s) = \sum_{j=1}^n \mathbf{x}_j(s) \mathbf{w}_j
\end{align*}
\begin{align*}
 = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - \mathbf{w}^T\mathbf{x}(S))^2]
\end{align*}
\begin{align*}
 = \alpha (v_{\pi}(S_t) - v_{\mathbf{w}}(S_t)) \mathbf{x}_t
\end{align*}
\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Objective function:

\begin{align*}
\Delta\mathbf{w}= \alpha (v_{\pi}(S) - v_{\mathbf{w}}(S)) \nabla_{\mathbf{w}}v_{\mathbf{w}}(S)
\end{align*}

Stochastic gradient descent:

\begin{align*}
\nabla_{\mathbf{w}}v_{\mathbf{w}}(S) = \mathbf{x}(S_t) = \mathbf{x}_t
\end{align*}
\begin{align*}
\mathbf{w} \leftarrow \mathbf{w} + \Delta \mathbf{w}
\end{align*}

Update = step-size x prediction error x feature vector

Linear Value Function Approximation

Approximate value function by a
linear combination of features

\begin{align*}
v_{\mathbf{w}}(s) = \mathbf{w}^T\mathbf{x}(s) = \sum_{j=1}^n \mathbf{x}_j(s) \mathbf{w}_j
\end{align*}
\begin{align*}
 = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - \mathbf{w}^T\mathbf{x}(S))^2]
\end{align*}
\begin{align*}
\nabla_{\mathbf{w}}v_{\mathbf{w}}(S) = \mathbf{x}(S_t) = \mathbf{x}_t
\end{align*}
\begin{align*}
 = \alpha (v_{\pi}(S_t) - v_{\mathbf{w}}(S_t)) \mathbf{x}_t
\end{align*}
\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Objective function:

\begin{align*}
\Delta\mathbf{w}= \alpha (v_{\pi}(S) - v_{\mathbf{w}}(S)) \nabla_{\mathbf{w}}v_{\mathbf{w}}(S)
\end{align*}

Stochastic gradient descent:

\begin{align*}
\mathbf{w} \leftarrow \mathbf{w} + \Delta \mathbf{w}
\end{align*}

Update = step-size x prediction error x feature vector

Linear Value Function Approximation

Approximate value function by a
linear combination of features

\begin{align*}
v_{\mathbf{w}}(s) = \mathbf{w}^T\mathbf{x}(s) = \sum_{j=1}^n \mathbf{x}_j(s) \mathbf{w}_j
\end{align*}
\begin{align*}
 = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - \mathbf{w}^T\mathbf{x}(S))^2]
\end{align*}
\begin{align*}
\nabla_{\mathbf{w}}v_{\mathbf{w}}(S) = \mathbf{x}(S_t) = \mathbf{x}_t
\end{align*}
\begin{align*}
 = \alpha (v_{\pi}(S_t) - v_{\mathbf{w}}(S_t)) \mathbf{x}_t
\end{align*}
\begin{align*}
 J(\mathbf{w}) = \mathbb{E}_{S \sim d}[(v_{\pi}(S) - v_{\mathbf{w}}(S))^2]
\end{align*}

Objective function:

\begin{align*}
\Delta\mathbf{w}= \alpha (v_{\pi}(S) - v_{\mathbf{w}}(S)) \nabla_{\mathbf{w}}v_{\mathbf{w}}(S)
\end{align*}

Stochastic gradient descent:

\begin{align*}
\mathbf{w} \leftarrow \mathbf{w} + \Delta \mathbf{w}
\end{align*}

Update = step-size x prediction error x feature vector

\begin{align*}
 = \alpha (v_{\pi}(S_t) - \mathbf{w}^T\mathbf{x}(s)) \mathbf{x}_t
\end{align*}

Linear Value Function Approximation for Prediction: MC

Monte-Carlo: is an unbiased sample of true value

Apply “supervised learning” to (online) “training data”:

\begin{align*}
\Delta\mathbf{w} = \alpha (v_{\pi}(S_t) - v_{\mathbf{w}}(S_t)) \mathbf{x}_t
\end{align*}
\begin{align*}
= \alpha (G_t - v_{\mathbf{w}}(S_t)) \mathbf{x}_t
\end{align*}

Linear Monte-Carlo evaluation converges to the global optimum

Even when using non-linear value function approximation it converges

(but perhaps to a local optimum)

Linear Value Function Approximation for Prediction: TD(0)

TD(0): is a biased sample of true value

Apply “supervised learning” to (online) “training data”:

\begin{align*}
\Delta\mathbf{w} = \alpha (v_{\pi}(S_t) - v_{\mathbf{w}}(S_t)) \mathbf{x}_t
\end{align*}

This is akin to a non-stationary regression problem

But it’s a bit different: the target depends on our parameters!

\begin{align*}
= \alpha (R_{t+1} + \gamma v_{\mathbf{w}}(S_{t+1}) - v_{\mathbf{w}}(S_t)) \mathbf{x}_t
\end{align*}
\begin{align*}
= \alpha \delta_t \mathbf{x}_t
\end{align*}

TD-error

Control with value-function approximation

Policy evaluation: Approximate policy evaluation,

Policy improvement: E.g., -greedy policy improvement

Action-Value Function Approximation

Approximate the action-value function

For instance, with linear function approximation with state-action features

\begin{align*}
q_{\mathbf{w}}(s,a) = \mathbf{x}(s,a)^T \mathbf{w}
\end{align*}

Stochastic gradient descent update

\begin{align*}
\Delta\mathbf{w} = \alpha (q_{\pi}(s,a) - q_{\mathbf{w}}(s,a)) \nabla_{\mathbf{w}}q_{\mathbf{w}}(s,a)
\end{align*}
\begin{align*}
 = \alpha (q_{\pi}(s,a) - q_{\mathbf{w}}(s,a)) \mathbf{x}(s,a)
\end{align*}

Action-Value Function Approximation

Approximate the action-value function

For instance, with linear function approximation with state-action features

\begin{align*}
q_{\mathbf{w}}(s,a) = \mathbf{x}(s,a)^T \mathbf{w}
\end{align*}

Stochastic gradient descent update

\begin{align*}
\Delta\mathbf{w} = \alpha (q_{\pi}(s,a) - q_{\mathbf{w}}(s,a)) \nabla_{\mathbf{w}}q_{\mathbf{w}}(s,a)
\end{align*}
\begin{align*}
 = \alpha (q_{\pi}(s,a) - q_{\mathbf{w}}(s,a)) \mathbf{x}(s,a)
\end{align*}

state

Action-Value Function Approximation

Approximate the action-value function

For instance, with linear function approximation with state-action features

\begin{align*}
q_{\mathbf{w}}(s,a) = \mathbf{x}(s,a)^T \mathbf{w}
\end{align*}

Stochastic gradient descent update

\begin{align*}
\Delta\mathbf{w} = \alpha (q_{\pi}(s,a) - q_{\mathbf{w}}(s,a)) \nabla_{\mathbf{w}}q_{\mathbf{w}}(s,a)
\end{align*}
\begin{align*}
 = \alpha (q_{\pi}(s,a) - q_{\mathbf{w}}(s,a)) \mathbf{x}(s,a)
\end{align*}

state

\begin{align*}
\mathbf{q}_{\mathbf{w}}(s) = \mathbf{W} \mathbf{x}(s)
\end{align*}
\begin{align*}
\mathbf{W} \in \mathbb{R}^{m \times n}
\end{align*}
\begin{align*}
\mathbf{x}(s) \in \mathbb{R}^n
\end{align*}
\begin{align*}
\mathbf{q} \in \mathbb{R}^m
\end{align*}
\begin{align*}
q_{\mathbf{w}}(s,a) = \mathbf{q}_{\mathbf{w}}(s)[a] = \mathbf{x}(s)^T  \mathbf{w}_a 
\end{align*}
\begin{align*}
\mathbf{w}_a = \mathbf{W}_a^T
\end{align*}

Action-Value Function Approximation

Approximate the action-value function

For instance, with linear function approximation with state-action features

\begin{align*}
q_{\mathbf{w}}(s,a) = \mathbf{x}(s,a)^T \mathbf{w}
\end{align*}

Stochastic gradient descent update

\begin{align*}
\Delta\mathbf{w} = \alpha (q_{\pi}(s,a) - q_{\mathbf{w}}(s,a)) \nabla_{\mathbf{w}}q_{\mathbf{w}}(s,a)
\end{align*}
\begin{align*}
 = \alpha (q_{\pi}(s,a) - q_{\mathbf{w}}(s,a)) \mathbf{x}(s,a)
\end{align*}

state

\begin{align*}
\mathbf{q}_{\mathbf{w}}(s) = \mathbf{W} \mathbf{x}(s)
\end{align*}
\begin{align*}
\mathbf{W} \in \mathbb{R}^{m \times n}
\end{align*}
\begin{align*}
\mathbf{x}(s) \in \mathbb{R}^n
\end{align*}
\begin{align*}
\mathbf{q} \in \mathbb{R}^m
\end{align*}
\begin{align*}
q_{\mathbf{w}}(s,a) = \mathbf{q}_{\mathbf{w}}(s)[a] = \mathbf{x}(s)^T  \mathbf{w}_a 
\end{align*}
\begin{align*}
\mathbf{w}_a = \mathbf{W}_a^T
\end{align*}
\begin{align*}
\Delta\mathbf{w}_a = \alpha (q_{\pi}(s,a) - q_{\mathbf{w}}(s,a)) \nabla_{\mathbf{w}}q_{\mathbf{w}}(s,a)
\end{align*}
\begin{align*}
 = \alpha (q_{\pi}(s,a) - q_{\mathbf{w}}(s,a)) \mathbf{x}(s)
\end{align*}
\begin{align*}
\Delta\mathbf{w}_b = 0, \quad \forall a \neq b
\end{align*}

Action-Value Function Approximation

Should we use action-in, or action-out?

Action in:

Action out:

\begin{align*}
q_{\mathbf{w}}(s,a) = \mathbf{x}(s,a)^T \mathbf{w}
\end{align*}
\begin{align*}
\mathbf{q}_{\mathbf{w}}(s) = \mathbf{W} \mathbf{x}(s) \text{ such that } q_{\mathbf{w}}(s,a) = \mathbf{q}_{\mathbf{w}}(s)[a]
\end{align*}

One reuses the same weights, the other the same features

Unclear which is better in general

If we want to use continuous actions, action-in is easier (later lecture on actor-critic)

For (small) discrete action spaces, action-out is common (e.g., DQN)

Convergence Questions

➢When do incremental prediction algorithms converge?

➢When using bootstrapping (i.e. TD)?

➢When using (e.g., linear) value function approximation?

➢When using off-policy learning?

​

➢Ideally, we would like algorithms that converge in all cases

​

➢Alternatively, we want to understand when algorithms do, or do not, converge

Example of divergence

x = 1

x = 2

r = 0

v = w

v = 2w

\begin{align*}
    =\mathbf{w}_t + \alpha_t ( 2\gamma -1 )\mathbf{w}_t
\end{align*}
\begin{align*}
\mathbf{w}_t > 0
\end{align*}
\begin{align*}
\gamma > 1/2
\end{align*}
\begin{align*}
\mathbf{w}_{t+1} > \mathbf{w}_t
\end{align*}
\begin{align*}
\lim_{t \rightarrow \infty} \mathbf{w}_t = \infty
\end{align*}
\begin{align*}
    = \mathbf{w}_t + \alpha_t (0 + \gamma 2\mathbf{w}_t - \mathbf{w}_t) 1 
\end{align*}
\begin{align*}
    = \mathbf{w}_t + \alpha_t (r + \gamma v(s' ) - v(s)) x(s) 
\end{align*}

Let’s use TD only on this transition.

\begin{align*}
    \mathbf{w}_{t+1} &= \mathbf{w}_t + \alpha_t (r + \gamma v(s' ) - v(s)) \nabla v(s) 
\end{align*}

Example of divergence

x = 1

x = 2

r = 0

v = w

v = 2w

Algorithms that combine

➢bootstrapping

➢off-policy learning

➢function approximation

… may diverge

Deadly triad

Convergence of Prediction Algorithms

On/Off-Policy

Algorithm

Table Lookup

Linear

Non-Linear

On-Policy

MC

TD

Off-Policy

MC

TD

Deadly triad

Bootstrapping

Off-policy

Function Approximation

Convergence of Control Algorithms

➢Tabular control learning algorithms (e.g., Q-learning) can be extended to FA

(e.g., Deep Q Network — DQN)

​

➢The theory of control with function approximation is not fully developed

​

➢Tracking is often preferred to convergence

(I.e., continually adapting the policy instead of converging to a fixed policy)

Experience Replay

Batch methods seek to find the best fitting value function for a given set of past experience (“training data")

Given experience consisting of trajectories of experience

\begin{align*}
\mathcal{D} = \{S_0, A_0, R_1, S_1, \dots, S_t \}
\end{align*}

1. Sample transition(s), e.g., for

2. Apply stochastic gradient descent update

\begin{align*}
\Delta \mathbf{w} = \alpha (R_{n+1} + \gamma v_{\mathbf{w}}(S_{n+1}) - v_{\mathbf{w}}(S_{n})) \nabla_{\mathbf{w}} v_{\mathbf{w}}(S_n)
\end{align*}

3. Can re-use old data

Repeat

This is a form of batch learning

Beware: the data may be off-policy if the policy changes!

Take home message

•For MDPs with large state and action spaces, VFA can be used.

​

•There are different classes of VFA: tabular, linear, and non-linear.

​

•Stochastic gradient descent can be used to approximate the true value function.

​

•Divergence can occur when combining:

•Bootstrapping

•Off-policy learning

•Function approximation

​

​

Take home message

•Further studying:

•Sections 9.1, 9.2, 9.3, 9.4, 9.5, 9.7, 10.1, 11.3 from the book Reinforcement Learning: An
Introduction

•Short notes: https://people.stfx.ca/jdelamer/courses/csci-531/lectures/rl/approximate-
solutions/on-policy-prediction.html

•Video (UCL x DeepMind)

•Lectures 7:
https://www.youtube.com/watch?v=ook46h2Jfb4&list=PLqYmG7hTraZCRwoyGxvQkqVrZgDQi4m-
5&index=7&t=13s

•Next lecture:

•Policy Gradient and Actor-Critic

​