Reinforcement Learning · Lecture 1

Exploration Methods

Rafael Fernandes Cunha

rafaelcunha2013.github.io

Welcome!

Rafael F. Cunha

Lecturer @ Leiden University

Ph.D. Candidate @ University of Groningen

​

Research topics:

•Multiagent RL 

•Transfer Learning in RL

•RL-based Post-training of LLMs with Analytic/Verifiable
Rewards 

​

MSc. in Systems and Control.

BSc. in Electronic Engineering.

​

Hobbies: Biking in the countryside.

Today’s topics

1.Present the course content and organization

2.Multi-armed Bandit Problem

3.Action-value methods

a.Greedy

b. -greedy

c.Upper Confidence Bound

d.SoftMax

4.Policy gradient methods

1.Describe the multi-armed Bandit Problem.

2.Explain the Exploitation x Exploration dilemma.

3.Identify the terms of an update rule.

4.Explain action-value methods.

5.Explain policy gradient methods.

Learning outcomes

Course Outline

No.

Topics

Ch.

1

Exploration Methods

2

2

The RL Problem and Finite Markov Decision Process

3

3

Dynamic Programming

4

4

Monte Carlo Methods, TD-learning and SARSA

5, 6

5

Q-learning, double q-learning, DQN

6

6

Function Approximators

9

7

Policy Gradients and Actor-Critics

13

8

Model Based RL

8

Course Material

​

•Main Material

•Book Reinforcement Learning: An Introduction

•Freely available at the authors website:
http://incompleteideas.net/book/the-book-2nd.html

​

•Other Materials

•UCL x Deepmind RL Lecture Series:
https://www.youtube.com/playlist?list=PLqYmG7hTraZCRwoyGxv
QkqVrZgDQi4m-5

•Lecture notes from the course CSCI 531 from the St. Francis
Xavier university.
https://people.stfx.ca/jdelamer/courses/csci-
531/index.html

•Gymnasium, a library with many environments to evaluate the RL
algorithms:
https://gymnasium.farama.org/

​

The multi-armed bandit problem

•You are in a casino with many slot machines.

•You will play 1000 times.

•Each time, you choose one machine.

•Your goal is to win as much money as possible.

How do you choose which slot
machine to play each time?

What do we know about these slot machines?

Consider the 1st slot machine.

You pull its arm 5 times and observe
the following payouts (rewards
).

= 2

= 1

= -1

= 3

= 0

Why are we receiving
different rewards if the
machine is the same?

Which mathematical concept are we talking about here? [Review]

1

Random variables (RVs)

•RVs are the staple of probability & statistics

​

Do you know what a random variable is?

It’s a variable whose value is unknown, not fixed, but dependent upon chance.

«Which is the area of a triangle
given that the base
is 12cm and
the height
is 14cm?»

Conventional
variable: its value
is fixed once we
define
and

«How tall is a random man from
the Netherlands?
»

«I am going to flip a fair coin.
Which is the result?
»

Height of Jan

Result of coin

Random
variables

[Review]

Two Flavors of RV: Discrete vs. Continuous

[Review]

Probability mass function (PMF)

Probability density function (PDF)

Every machine has a hidden “personality” (Its PDF)

1

Distribution on rewards

\begin{align*}
     \mathcal{R}_1
\end{align*}

•Represents the true, hidden probability of all possible rewards for a single action.

•The area under a section of the curve gives the probability of a reward falling within that range.

​

What do we know about these slot machines?

1

Consider the 1st slot machine.

You pull its arm 5 times and observe
the following payouts (rewards
).

= 2

= 1

= -1

= 3

= 0

Why are we receiving
different rewards if the
machine is the same?

Rewards are sampled
from a distribution!

Reward distribution - Machine 1

\begin{align*}
    R_t \sim  \mathcal{R}_1
\end{align*}

What does sampling from a distribution mean? [Review]

Sampling from a probability distribution

•Sampling from a probability distribution = generate an individual/a data point according to that distribution

•As we collect more samples, the shape of our data converges to the hidden truth.

[Review]

What do we know about these slot machines?

1

Consider the 1st slot machine.

You pull its arm 5 times and observe
the following payouts (rewards
).

= 2

= 1

= -1

= 3

= 0

Why are we receiving
different rewards if the
machine is the same?

What value would you guess for ?

Rewards are sampled
from a distribution!

Reward distribution - Machine 1

The sample mean approximates the true mean.

\begin{align*}
    R_t \sim  \mathcal{R}_1
\end{align*}

What is Expected value? [Review]

Expected Value

We throw a fair dice and observe 2, 1, 3.

Is this a good summary statistic of the dice?

\begin{align*}
\overline{X} = \frac{2+1+3}{3} = 2
\end{align*}

[Review]

Rolling a Dice

\begin{align*}
\mathbb{E}(X) = 1\cdot\frac{1}{6} + 2\cdot\frac{1}{6} + 3\cdot\frac{1}{6} + 4\cdot\frac{1}{6} + 5\cdot\frac{1}{6} + 6\cdot\frac{1}{6} = 3.5
\end{align*}

The expected value of an RV is a constant:

\begin{align*}
\overline{X_3} = \frac{1}{3}(1+2+3) = 2
\end{align*}

The sample mean depends on the data:

[Review]

Expected Value

The sample mean

is very different from the expected value

\begin{align*}
\overline{X}_N = \frac{1}{N}\sum_{i=1}^{N} X_i
\end{align*}
\begin{align*}
\mathbb{E}(X) = \sum_{x} x\; f(x)
\end{align*}
\begin{align*}
\overline{X}_N \longrightarrow \mathbb{E}(X)
\end{align*}

Only in the long run,

[Review]

Throwing a Coin

Number of trials

Value

[Review]

Action value estimates

Action 1

Action 2

Action 3

Action 4

Estimated value of action at time step

Which machine would you
decide to choose next?

​

Why?

Highest estimate

Highest uncertainty

Reward Distributions – 4 Bandits (slot-machines)

Which machine would you
decide to choose next?

​

Why?

Highest true mean!

Expected reward of action a

\begin{align*}
q_*(a) &= \mathbb{E}[R_t | A_t = a] 
\end{align*}

Action at time t

Reward Distributions – 4 Bandits (slot-machines)

Which machine would you
decide to choose next?

​

Why?

Highest true mean!

Problem:

We do not know the reward distributions a priori!

Expected reward of action a

\begin{align*}
q_*(a) &= \mathbb{E}[R_t | A_t = a] 
\end{align*}

Action at time t

Exploration-Exploitation Dilemma

Exploitation: Choose the machine with the highest estimated reward.

Exploration: Try other machines to learn more about their rewards.

This problem appears in many real systems:

•Netflix recommendations

•TikTok video selection

•Online advertising

Why should we care?

How do we balance these two goals?

There are other options, but first, let’s formalize the multi-armed bandit problem.

Formalizing the Multi-Armed Bandit problem

➢Multi-armed bandit: Set of distributions

​

​

​

Set of actions

Distribution on rewards, given action

\begin{align*}
\{\mathcal{R}_a | a \in \mathcal{A}\}
\end{align*}

Formalizing the Multi-Armed Bandit problem

➢Multi-armed bandit: Set of distributions

​

​

​

➢At each step , the agent selects an action

Why write instead of ?

\begin{align*}
\{\mathcal{R}_a | a \in \mathcal{A}\}
\end{align*}
\begin{align*}
A_t \in \mathcal{A}
\end{align*}

is a random variable

is one fixed action

Formalizing the Multi-Armed Bandit problem

➢Multi-armed bandit: Set of distributions

​

​

​

➢At each step , the agent selects an action

➢The environment generates a reward

\begin{align*}
\{\mathcal{R}_a | a \in \mathcal{A}\}
\end{align*}
\begin{align*}
A_t \in \mathcal{A}
\end{align*}
\begin{align*}
 R_t \sim  \mathcal{R}_{A_t}
\end{align*}

Goal: Maximize cumulative reward

\begin{align*}
\sum_{i=1}^t R_t 
\end{align*}

To achieve the goal, we learn a policy : a probability distribution on

\begin{align*}
\mathcal{A}
\end{align*}

Policy

Formally

\begin{align*}
\pi : \mathcal{A} \rightarrow [0, 1]
\end{align*}

Why it needs to be between 0 and 1? [Review]

To achieve the goal, we learn a policy : a probability distribution on

\begin{align*}
\mathcal{A}
\end{align*}

Probability Mass Function (PMF)

[Review]

•A function is a valid PMF if it follows two simple rules:

​

1.Rule 1: All outputs are valid probabilities.

​

​

​

2.Rule 2: The total probability is 1.

​

​

​

\begin{align*}
\mathbb{P}(X = H) = \mathbb{P}(X = T) = 0.5
\end{align*}
\begin{align*}
f_X(x) = \mathbb{P}(X = x)
\end{align*}
\begin{align*}
X \sim f_X
\end{align*}

•A PMF describes the probability for each possible outcome of a
discrete random variable.

A Policy is a PMF over actions

•A policy is simply a PMF where the "outcomes" are
the possible
actions the agent can take.

​

•A policy must follow the same two rules:

​

Rule 1 Applied to Policies: The probability of selecting any
single action `a` must be between 0 and 1.

​

​

​

​

Rule 2 Applied to Policies: The probabilities of selecting all
possible actions must sum to 1 (the agent has to choose

something
).

Policy

Action

1

2

3

4

5

6

7

8

9

10

Deterministic

0

0

1

0

0

0

0

0

0

0

Stochastic

0.05

0.01

0.50

0.13

0.13

0.01

0.10

0.01

0.05

0.01

Deterministic X Stochastic policy

To achieve the goal, we learn a policy : a distribution on

\begin{align*}
\mathcal{A}
\end{align*}

Formally

\begin{align*}
\pi : \mathcal{A} \rightarrow [0, 1]
\end{align*}
\begin{align*}
\pi(a) = \text{Pr}\{A_t = a\}\text{, for all } a \in \mathcal{A}
\end{align*}
\begin{align*}
\sum_{a=1}^{10} \pi(a) = 1
\end{align*}

Policy

Action

1

2

3

4

5

6

7

8

9

10

Deterministic

0

0

1

0

0

0

0

0

0

0

Stochastic

0.05

0.01

0.50

0.13

0.13

0.01

0.10

0.01

0.05

0.01

Deterministic X Stochastic policy

\begin{align*}
\pi : \mathcal{A} \rightarrow [0, 1]
\end{align*}

How to find the optimal policy?

Action-Value Methods

➢First: Learn the reward distribution, or just its mean (action values)

​

➢Second: Find a policy based on the action values

■Greedy

■-greedy

■Upper Confidence Bound (UCB)

■Softmax policy

Action-Value function

To achieve the goal, we learn a policy : a distribution on

\begin{align*}
\mathcal{A}
\end{align*}
\begin{align*}
q_*(a) &= \mathbb{E}[R_t | A_t = a] 
\end{align*}
\begin{align*}
\{\mathcal{R}_a | a \in \mathcal{A}\}
\end{align*}

Action-Value Methods

The action-value function for action is the expected reward

\begin{align*}
q_*(a) &= \mathbb{E}[R_t | A_t = a] 
\end{align*}

The optimal action is

\begin{align*}
a^* = \arg \max_{a \in \mathcal{A}} q_*(a)
\end{align*}

The Regret of an optimal action is zero.

The Regret of an action is

\begin{align*}
\Delta_a = q_*(a^*) - q_*(a)
\end{align*}

Sample-average method

How to compute the action-value function?

Weight-average method

?

?

Sample-Average Method: Direct Average

Time t

Action 1

Action 2

Action 3

t = 1

t = 2

t = 3

t = 4

t = 5

t = 6

\begin{align*}
Q_5(1) = \frac{R_1 \cdot 1 + R_2 \cdot 0 + R_3 \cdot 1 + R_4 \cdot 0}{1 + 0 + 1 + 0}
\end{align*}
\begin{align*}
Q_5(2) = \frac{R_1 \cdot 0 + R_2 \cdot 1 + R_3 \cdot 0 + R_4 \cdot 0}{0 + 1 + 0 + 0}
\end{align*}
\begin{align*}
Q_t(a) \doteq \frac{\sum_{i=1}^{t-1} R_i \cdot \mathbb{I}(A_i = a)}{n_t(a)}
\end{align*}
\begin{align*}
n_t(a) \doteq \sum_{i=1}^{t-1} \mathbb{I}(A_i = a)
\end{align*}

Indicator function:

\begin{align*}
     \mathbb{I}(True) = 1 \\
     \mathbb{I}(False) = 0
\end{align*}
\begin{align*}
    Q_t(a) \doteq \frac{\text{sum of rewards when }a \text{ taken prior to }t } {\text{number of times } a \text{ taken prior to }t} 
\end{align*}

Sampling Rewards

Sample-Average method: Incremental Implementation

\begin{align*}
Q_{n+1} &= \frac{1}{n}\sum_{i=1}^{n} R_i \\
&= \frac{1}{n}\left(R_n + \sum_{i=1}^{n-1} R_i\right) \\
&= \frac{1}{n}\left(R_n + (n-1)\frac{1}{n-1}\sum_{i=1}^{n-1} R_i\right) \\
&= \frac{1}{n}\Big(R_n + (n-1)Q_n\Big) \\
&= \frac{1}{n}\Big(R_n + nQ_n - Q_n\Big) \\
&= Q_n + \frac{1}{n}\Big[R_n - Q_n\Big],
\end{align*}

Consider only one action

: reward received after the th selection of this action

: Estimate of its action value after it has been selected times.

\begin{align*}
Q_{n+1} &= \frac{1}{n}\sum_{i=1}^{n} R_i \\
&= \frac{1}{n}\left(R_n + \sum_{i=1}^{n-1} R_i\right) \\
&= \frac{1}{n}\left(R_n + (n-1)\frac{1}{n-1}\sum_{i=1}^{n-1} R_i\right) \\
&= \frac{1}{n}\Big(R_n + (n-1)Q_n\Big) \\
&= \frac{1}{n}\Big(R_n + nQ_n - Q_n\Big) \\
&= Q_n + \frac{1}{n}\Big[R_n - Q_n\Big],
\end{align*}
\begin{align*}
Q_{n+1} &= \frac{1}{n}\sum_{i=1}^{n} R_i \\
&= \frac{1}{n}\left(R_n + \sum_{i=1}^{n-1} R_i\right) \\
&= \frac{1}{n}\left(R_n + (n-1)\frac{1}{n-1}\sum_{i=1}^{n-1} R_i\right) \\
&= \frac{1}{n}\Big(R_n + (n-1)Q_n\Big) \\
&= \frac{1}{n}\Big(R_n + nQ_n - Q_n\Big) \\
&= Q_n + \frac{1}{n}\Big[R_n - Q_n\Big],
\end{align*}
\begin{align*}
Q_{n+1} &= \frac{1}{n}\sum_{i=1}^{n} R_i \\
&= \frac{1}{n}\left(R_n + \sum_{i=1}^{n-1} R_i\right) \\
&= \frac{1}{n}\left(R_n + (n-1)\frac{1}{n-1}\sum_{i=1}^{n-1} R_i\right) \\
&= \frac{1}{n}\Big(R_n + (n-1)Q_n\Big) \\
&= \frac{1}{n}\Big(R_n + nQ_n - Q_n\Big) \\
&= Q_n + \frac{1}{n}\Big[R_n - Q_n\Big],
\end{align*}
\begin{align*}
Q_{n+1} &= \frac{1}{n}\sum_{i=1}^{n} R_i \\
&= \frac{1}{n}\left(R_n + \sum_{i=1}^{n-1} R_i\right) \\
&= \frac{1}{n}\left(R_n + (n-1)\frac{1}{n-1}\sum_{i=1}^{n-1} R_i\right) \\
&= \frac{1}{n}\Big(R_n + (n-1)Q_n\Big) \\
&= \frac{1}{n}\Big(R_n + nQ_n - Q_n\Big) \\
&= Q_n + \frac{1}{n}\Big[R_n - Q_n\Big],
\end{align*}
\begin{align*}
Q_{n+1} &= \frac{1}{n}\sum_{i=1}^{n} R_i \\
&= \frac{1}{n}\left(R_n + \sum_{i=1}^{n-1} R_i\right) \\
&= \frac{1}{n}\left(R_n + (n-1)\frac{1}{n-1}\sum_{i=1}^{n-1} R_i\right) \\
&= \frac{1}{n}\Big(R_n + (n-1)Q_n\Big) \\
&= \frac{1}{n}\Big(R_n + nQ_n - Q_n\Big) \\
&= Q_n + \frac{1}{n}\Big[R_n - Q_n\Big],
\end{align*}

Extending to multiple actions

\begin{align*}
Q_{n+1}(a) = Q_n(a) + \alpha_n(a) \Big[ R_n - Q_n(a) \Big]
\end{align*}
\begin{align*}
    Q_n \doteq \frac{R_1 + R_2 + \cdots + R_{n-1}}{n-1} 
\end{align*}
\begin{align*}
    Q_{n+1} = Q_n + \frac{1}{n}\Big[ R_n - Q_n \Big] 
\end{align*}
\begin{align*}
    \frac{1}{n_t(a)} 
\end{align*}
\begin{align*}
    n_t(a) \doteq \sum_{i=1}^{t-1} \mathbb{I}(A_i = a) 
\end{align*}
\begin{align*}
    \text{NewEstimate} \leftarrow \text{OldEstimate} + \text{StepSize} \Big [ \text{Target} - \text{OldEstimate} \Big ] 
\end{align*}

Error in the estimate!

Reduced by taking a step toward the “target”

Sample-Average method: Incremental Implementation

\begin{align*}
Q_{n+1}(a) = Q_n(a) + \alpha_n(a) \Big[ R_n - Q_n(a) \Big]
\end{align*}

Sample-average method is appropriate for stationary bandit problems

reward probability distribution do not
change
over time

Variable StepSize

Stationary bandit problems

\begin{align*}
Q_{n+1}(a) = Q_n(a) + \alpha_n(a) \Big[ R_n - Q_n(a) \Big]
\end{align*}

What are the properties of this variable step size? [Review]

Why the Step-Size Guarantees Learning

[Review]

The Step-Size forms a sequence.

​

A series is the sum of that sequence.

\begin{align*}
\sum_{n=1}^{\infty} \frac{1}{n^p} \;\text{converges iff}\; p > 1
\end{align*}

Sample-average method

(more suitable for stationary bandit)

How to compute the action-value function?

Weight-average method

\begin{align*}
\alpha_n(a) = \frac{1}{n_t(a)}
\end{align*}
\begin{align*}
Q_{n+1}(a) = Q_n(a) + \alpha_n(a) \Big[ R_n - Q_n(a) \Big]
\end{align*}

?

Nonstationary
problems

Give more weight to
recent rewards
than to
long-past rewards

Use constant StepSize
parameter!

\begin{align*}
Q_{n+1}(a) = Q_n(a) + \alpha\Big[R_n - Q_n(a)\Big]
\end{align*}

What means ?

Tracking a Nonstationary Problem (Weight-average method)

Nonstationary
problems

Give more weight to
recent rewards
than to
long-past rewards

Use constant StepSize
parameter!

\begin{align*}
Q_{n+1}(a) = Q_n(a) + \alpha\Big[R_n - Q_n(a)\Big]
\end{align*}

Tracking a Nonstationary Problem (Weight-average method)

From stochastic
approximation theory

Conditions required to
assure convergence
with probability 1

Does a constant satisfy
these conditions?

Steps are large enough to overcome any
initial conditions or random fluctuations

\begin{align*}
    \sum_{n=1}^{\infty} \alpha_n = \infty
\end{align*}

Steps become small enough to
assure convergence

\begin{align*}
    \sum_{n=1}^{\infty} \alpha_n^2 < \infty
\end{align*}

Nonstationary
problems

Give more weight to
recent rewards
than to
long-past rewards

Use constant StepSize
parameter!

\begin{align*}
Q_{n+1}(a) = Q_n(a) + \alpha\Big[R_n - Q_n(a)\Big]
\end{align*}

Tracking a Nonstationary Problem (Weight-average method)

Which satisfies
these conditions?

\begin{align*}
\alpha_n(a) = \frac{1}{n_t(a)}
\end{align*}

Sample-average method

\begin{align*}
\alpha_n(a) = \alpha
\end{align*}

Weight-average method

Estimates never completely converge but

continue to vary in response to the most
recently received rewards.

Desirable in a
nonstationary
environment!

Usually the case in
Reinforcement
Learning problems.

\begin{align*}
    \sum_{n=1}^{\infty} \alpha_n = \infty
\end{align*}
\begin{align*}
    \sum_{n=1}^{\infty} \alpha_n^2 < \infty
\end{align*}

Nonstationary
problems

Give more weight to
recent rewards
than to
long-past rewards

Use constant StepSize
parameter!

\begin{align*}
Q_{n+1}(a) = Q_n(a) + \alpha\Big[R_n - Q_n(a)\Big]
\end{align*}

Tracking a Nonstationary Problem (Weight-average method)

\begin{align*}
Q_{n+1}(a) = (1-\alpha)^n Q_1(a) + \sum_{i=1}^{n} \alpha(1-\alpha)^{n-i} R_i
\end{align*}

More details in
book’s page 32.

Nonstationary
problems

Give more weight to
recent rewards
than to
long-past rewards

Use constant StepSize
parameter!

Tracking a Nonstationary Problem (Weight-average method)

\begin{align*}
Q_{n+1}(a) = (1-\alpha)^n Q_1(a) + \sum_{i=1}^{n} \alpha(1-\alpha)^{n-i} R_i
\end{align*}

Initial estimate

Past rewards

weighted average

Sample-average method

(more suitable for stationary bandit)

How to compute the action-value function?

Weight-average method

(suitable for stationary and nonstationary bandit)

\begin{align*}
\alpha_n(a) = \frac{1}{n_t(a)}
\end{align*}
\begin{align*}
Q_{n+1}(a) = Q_n(a) + \alpha_n(a) \Big[ R_n - Q_n(a) \Big]
\end{align*}
\begin{align*}
\alpha_n(a) = \alpha
\end{align*}
\begin{align*}
Q_{n+1}(a) = (1- \alpha)^n Q_1(a) + \sum_{i=1}^n \alpha (1-\alpha)^{n-i} R_i
\end{align*}

Update rule

NewEstimate

OldEstimate

StepSize

InformativeValue

Target – OldEstimate

Error

Gradient

Learning Rate

Fixed or variable

Algorithms: Action-value methods

Greedy

UCB

Softmax

-greedy

Action-value update rule

Policies

\begin{align*}
Q_{n+1}(a) = Q_n(a) + \alpha_n(a) \Big[ R_n - Q_n(a) \Big]
\end{align*}

Algorithms: Action-value methods

Greedy

UCB

Softmax

?

?

?

?

-greedy

Greedy

Select the action with the highest estimated value

(It always exploits)

Problems?

Suppose that our current estimates are
represented by

We would keep selecting action 4 for how long?

In which suboptimal will we get stuck?

Does this induce a
deterministic or
stochastic policy?

\begin{align*}
  A_t = \underset{a \in \mathcal{A}}{\arg \max} \; Q_t(a)
\end{align*}
\begin{align*}
 \pi_t(a) =
    \begin{cases}
      1 \hspace{2mm} \text{if} \hspace{2mm} a=\underset{a \in \mathcal{A}}{\arg \max} \; Q_t(a)\\
       0 \hspace{7mm} \text{otherwise}
    \end{cases} 
\end{align*}

Greedy

Select the action with the highest estimated value

Problems?

Suppose that our current estimates are
represented by

It’s an estimated value! It can be wrong.

We will get stuck in a suboptimal forever!

\begin{align*}
  A_t = \underset{a \in \mathcal{A}}{\arg \max} \; Q_t(a)
\end{align*}

Greedy

Select the action with the highest estimated value

Problems?

It’s an estimated value! It can be wrong.

We will get stuck in a suboptimal forever!

Suppose that our current estimates are
represented by

Suboptimal: Action 5

Optimal action: Action 3

\begin{align*}
  A_t = \underset{a \in \mathcal{A}}{\arg \max} \; Q_t(a)
\end{align*}

Greedy

Select the action with the highest estimated value

Problems?

It’s an estimated value! It can be wrong.

We will get stuck in a suboptimal forever!

Suppose that our current estimates are
represented by

We need to find a way to keep updating the
action value estimates of the other actions!

\begin{align*}
  A_t = \underset{a \in \mathcal{A}}{\arg \max} \; Q_t(a)
\end{align*}

Greedy: Example

Most likely solution

Less likely solution

Which action is most likely to keep being selected in the long run?

Algorithms: Action-value methods

Greedy

UCB

Softmax

?

?

?

-greedy

\begin{align*}
 \pi_t(a) =
    \begin{cases}
      1 \hspace{2mm} \text{if} \hspace{2mm} a=\underset{a \in \mathcal{A}}{\arg \max} \; Q_t(a)\\
       0 \hspace{7mm} \text{otherwise}
    \end{cases} 
\end{align*}

-greedy

Small number

Is this expression representing
the above sentences?

➢Behave greedily with probability

​

​

➢Take a random action with probability

(It keeps exploring!)

\begin{align*}
       \pi_t(a) =
    \begin{cases}
      (1-\epsilon)  \hspace{2mm} \text{if} \hspace{2mm} a=\underset{a \in \mathcal{A}}{\arg \max} \; Q_t(a)\\
       \epsilon/|\mathcal{A}|  \hspace{8mm} \text{otherwise}
    \end{cases}  
\end{align*}
\begin{align*}
    A_t = \underset{ a \in \mathcal{A}}{\arg \max} \;Q_t(a) 
\end{align*}

-greedy

Small number

➢Behave greedily with probability

​

​

➢Take a random action with probability

(It keeps exploring!)

\begin{align*}
    A_t = \underset{ a \in \mathcal{A}}{\arg \max} \;Q_t(a) 
\end{align*}

Example:

\begin{align*}
    \mathcal{A} = \{1, 2, 3, 4\} 
\end{align*}
\begin{align*}
    \text{Cardinality of } \mathcal{A} \doteq |\mathcal{A}| = 4 
\end{align*}
\begin{align*}
    \epsilon = 0.4 
\end{align*}

Actions

1

2

3

4

Action-value estimates

3

1

4

-2

Probabilities

Exploit ()

Explore

(Uniformly random)

𝜖-greedy policy

\begin{align*}
    Q_t(a) 
\end{align*}
\begin{align*}
    \pi(a) 
\end{align*}

-greedy

Small number

➢Behave greedily with probability

​

​

➢Take a random action with probability

(It keeps exploring!)

\begin{align*}
    A_t = \underset{ a \in \mathcal{A}}{\arg \max} \;Q_t(a) 
\end{align*}

Example:

\begin{align*}
    \mathcal{A} = \{1, 2, 3, 4\} 
\end{align*}
\begin{align*}
    \text{Cardinality of } \mathcal{A} \doteq |\mathcal{A}| = 4 
\end{align*}
\begin{align*}
    \epsilon = 0.4 
\end{align*}

Actions

1

2

3

4

Action-value estimates

3

1

4

-2

Probabilities

Exploit ()

0

0

0.6

0

Explore

(Uniformly random)

0.1

0.1

0.1

0.1

𝜖-greedy policy

\begin{align*}
    Q_t(a) 
\end{align*}
\begin{align*}
    \pi(a) 
\end{align*}

-greedy

Small number

➢Behave greedily with probability

​

​

➢Take a random action with probability

(It keeps exploring!)

\begin{align*}
    A_t = \underset{ a \in \mathcal{A}}{\arg \max} \;Q_t(a) 
\end{align*}

Example:

\begin{align*}
    \mathcal{A} = \{1, 2, 3, 4\} 
\end{align*}
\begin{align*}
    \text{Cardinality of } \mathcal{A} \doteq |\mathcal{A}| = 4 
\end{align*}
\begin{align*}
    \epsilon = 0.4 
\end{align*}

Actions

1

2

3

4

Action-value estimates

3

1

4

-2

Probabilities

Exploit ()

0

0

0.6

0

Explore

(Uniformly random)

0.1

0.1

0.1

0.1

𝜖-greedy policy

0.1

0.1

0.7

0.1

\begin{align*}
    Q_t(a) 
\end{align*}
\begin{align*}
    \pi(a) 
\end{align*}
\begin{align*}
    \epsilon / |\mathcal{A}| 
\end{align*}
\begin{align*}
    (1 - \epsilon) + \epsilon/|\mathcal{A}| 
\end{align*}
\begin{align*}
    \epsilon / |\mathcal{A}| 
\end{align*}
\begin{align*}
    \epsilon / |\mathcal{A}| 
\end{align*}

-greedy

Small number

Is this expression representing
the above sentences?

➢Behave greedily with probability

​

​

➢Take a random action with probability

(It keeps exploring!)

\begin{align*}
       \pi_t(a) =
    \begin{cases}
      (1-\epsilon)  \hspace{2mm} \text{if} \hspace{2mm} a=\underset{a \in \mathcal{A}}{\arg \max} \; Q_t(a)\\
       \epsilon/|\mathcal{A}|  \hspace{8mm} \text{otherwise}
    \end{cases}  
\end{align*}
\begin{align*}
    A_t = \underset{ a \in \mathcal{A}}{\arg \max} \;Q_t(a) 
\end{align*}
\begin{align*}
     \pi_t(a) =
    \begin{cases}
      (1-\epsilon) + \epsilon/|\mathcal{A}| \hspace{2mm} \text{if} \hspace{2mm} a=\underset{a \in \mathcal{A}}{\arg \max} \; Q_t(a)\\
       \epsilon/|\mathcal{A}|  \hspace{18mm} \text{otherwise}
    \end{cases}  
\end{align*}

-greedy policy

\begin{align*}
\pi_t(a) &= \begin{cases} (1-\epsilon) + \epsilon/|\mathcal{A}| & \text{if } \; a = \underset{a\in\mathcal{A}}{\arg\max}\, Q_t(a) \\[1.5ex] \epsilon/|\mathcal{A}| & \qquad\qquad\quad\;\; \text{otherwise} \end{cases}
\end{align*}
\begin{align*}
\sum_{a} \pi_t(a) &= (1-\epsilon) + \frac{\epsilon}{|\mathcal{A}|} + (|\mathcal{A}|-1)\frac{\epsilon}{|\mathcal{A}|} \\
&= 1 - \epsilon + \frac{\epsilon}{|\mathcal{A}|} + \epsilon - \frac{\epsilon}{|\mathcal{A}|} \\
&= 1
\end{align*}

Rule 1: All outputs are valid probabilities.

Rule 2: The total probability is 1.

?

?

Is the -greedy policy a pmf?

Greedy x e-greedy

Average performance of -greedy action-
value methods on the 10-armed testbed.

A simple bandit algorithm (e-greedy)

A simple bandit algorithm (e-greedy)

Is this a good initialization? Is there a way that improves exploration?

Using Optimistic Initial Values

Greedy algorithm

Initializing action-values as 0

Initializing action-values optimistically

What is the maximum value that can be
obtained in a normal distribution?

Using Optimistic Initial Values

Greedy algorithm

Suppose action 9 is selected.

Initializing action-values as 0

Initializing action-values optimistically

Using Optimistic Initial Values

Greedy algorithm

Greedy will keep selecting action 9

Greedy will select all other actions several
times before the value estimates converge!

Initializing action-values as 0

Initializing action-values optimistically

Using Optimistic Initial Values

Greedy algorithm

Greedy will keep selecting action 9

Initializing action-values as 0

Initializing action-values optimistically

Greedy will select all other actions several
times before the value estimates converge!

Using Optimistic Initial Values: Example

Effect of using Optimistic Initial Values

\begin{align*}
\pi_t(a) = \begin{cases} (1-\epsilon) + \epsilon/|\mathcal{A}| & \text{if }\; a = \displaystyle\operatorname*{arg\,max}_{a\in\mathcal{A}} Q_t(a) \\ \epsilon/|\mathcal{A}| & \qquad\quad\text{otherwise} \end{cases}
\end{align*}

Algorithms: Action-value methods

Greedy

UCB

Softmax

?

?

-greedy

\begin{align*}
 \pi_t(a) =
    \begin{cases}
      1 \hspace{2mm} \text{if} \hspace{2mm} a=\underset{a \in \mathcal{A}}{\arg \max} \; Q_t(a)\\
       0 \hspace{7mm} \text{otherwise}
    \end{cases} 
\end{align*}

Optimism in the Face of Uncertainty

Up to now, we are considering only the action-value estimates to derive a policy.

3

​

2

​

1

​

0

​

-1

​

-2

​

-3

123

Action

Estimated
action values

\begin{align*}
Q(a_1)
\end{align*}
\begin{align*}
Q(a_3)
\end{align*}

What if we could consider the
uncertainty of our estimates?

\begin{align*}
Q(a_2)
\end{align*}

Optimism in the Face of Uncertainty

Up to now, we are considering only the action-value estimates to derive a policy.

3

​

2

​

1

​

0

​

-1

​

-2

​

-3

123

Action

Estimated
action values

\begin{align*}
Q(a_1)
\end{align*}
\begin{align*}
Q(a_2)
\end{align*}
\begin{align*}
Q(a_3)
\end{align*}

What if we could consider the
uncertainty of our estimates?

\begin{align*}
U(a_1)
\end{align*}
\begin{align*}
U(a_2)
\end{align*}
\begin{align*}
U(a_3)
\end{align*}

Uncertainty in the
estimation of

\begin{align*}
Q(a_2)
\end{align*}

Optimism in the Face of Uncertainty

Up to now, we are considering only the action value estimates to derive a policy.

3

​

2

​

1

​

0

​

-1

​

-2

​

-3

123

Action

Estimated
action values

\begin{align*}
Q(a_1)
\end{align*}
\begin{align*}
Q(a_2)
\end{align*}
\begin{align*}
Q(a_3)
\end{align*}

What if we could consider the
uncertainty of our estimates?

\begin{align*}
U(a_1)
\end{align*}
\begin{align*}
U(a_2)
\end{align*}
\begin{align*}
U(a_3)
\end{align*}

Uncertainty in the
estimation of

\begin{align*}
Q(a_2)
\end{align*}

Acting greedy with respect to the
action-value
results in action 1.

\begin{align*}
A_t = \arg\max_{a\in\mathcal{A}} Q_t(a)
\end{align*}

Acting greedy with respect to the
action-value + uncertainty
results
in action 2!

\begin{align*}
A_t = \operatorname*{arg\,max}_{a\in\mathcal{A}}\;\big(Q_t(a) + U_t(a)\big)
\end{align*}

Optimism in the Face of Uncertainty

Up to now, we are considering only the action value estimates to derive a policy.

3

​

2

​

1

​

0

​

-1

​

-2

​

-3

123

Action

Estimated
action values

\begin{align*}
Q(a_1)
\end{align*}
\begin{align*}
Q(a_2)
\end{align*}
\begin{align*}
Q(a_3)
\end{align*}
\begin{align*}
U(a_1)
\end{align*}
\begin{align*}
U(a_2)
\end{align*}
\begin{align*}
U(a_3)
\end{align*}

Acting greedy with respect to the
action-value + uncertainty
results
in action 2!

\begin{align*}
A_t = \operatorname*{arg\,max}_{a\in\mathcal{A}}\;\big(Q_t(a) + U_t(a)\big)
\end{align*}

After selecting action 2, we got new
information and reduce our uncertainty
and update the action-value estimate.

How can we compute this uncertainty?

Which property should it have?

Upper Confidence Bound (UCB)

We want to estimate an upper confidence
bound
for every action value

\begin{align*}
q_*(a) \le Q_t(a) + U_t(a)
\end{align*}

3

​

2

​

1

​

0

​

-1

​

-2

​

-3

a

Action

Estimated
action values

\begin{align*}
Q(a)
\end{align*}

Estimate

\begin{align*}
q_*(a)
\end{align*}

True Value

Upper confidence
bound

\begin{align*}
U(a)
\end{align*}

Uncertainty

\begin{align*}
A_t \doteq \operatorname*{arg\,max}_{a\in\mathcal{A}}\;\left[Q_t(a) + c\sqrt{\frac{\ln(t)}{n_t(a)}}\,\right]
\end{align*}

Upper Confidence Bound (UCB)

\begin{align*}
A_t = \operatorname*{arg\,max}_{a\in\mathcal{A}}\;\big(Q_t(a) + U_t(a)\big)
\end{align*}

Increases with time,
increasing uncertainty

scales the
uncertainty! Regulates the
amount of exploration.

Increases when action
a is selected,
decreasing uncertainty.

\begin{align*}
n_t(a) \doteq \sum_{i=1}^{t-1} \mathbb{I}(A_i = a)
\end{align*}

Upper Confidence Bound (UCB)

UCB generally performs better than -greedy on the multi-armed bandit problem.

\begin{align*}
\pi_t(a) = \begin{cases} 1 \;\;\text{if}\;\; a = \displaystyle\operatorname*{arg\,max}_{a\in\mathcal{A}}\;\left[Q_t(a) + c\sqrt{\tfrac{\ln(t)}{n_t(a)}}\,\right] \\[1ex] 0 \qquad\quad \text{otherwise} \end{cases}
\end{align*}
\begin{align*}
\pi_t(a) = \begin{cases} (1-\epsilon) + \epsilon/|\mathcal{A}| & \text{if }\; a = \displaystyle\operatorname*{arg\,max}_{a\in\mathcal{A}} Q_t(a) \\ \epsilon/|\mathcal{A}| & \qquad\quad\text{otherwise} \end{cases}
\end{align*}

Algorithms: Action-value methods

Greedy

UCB

Softmax

?

-greedy

\begin{align*}
 \pi_t(a) =
    \begin{cases}
      1 \hspace{2mm} \text{if} \hspace{2mm} a=\underset{a \in \mathcal{A}}{\arg \max} \; Q_t(a)\\
       0 \hspace{7mm} \text{otherwise}
    \end{cases} 
\end{align*}

Softmax Action Selection policy

It is possible to produce a stochastic policy from the action-value estimates.

Taking the greedy action is not the only possibility!

Temperature.

High values => actions (nearly) equiprobable.

Low values => the value estimator has more weight.

\begin{align*}
 \pi_t(a) = \frac{e^{Q_t(a)/\tau}}{\sum_{b} e^{Q_t(b)/\tau} } 
\end{align*}

Softmax Action Selection policy

Is the softmax action selection policy a pmf?

Rule 1: All outputs are valid probabilities.

Rule 2: The total probability is 1.

?

?

\begin{align*}
 \pi_t(a) = \frac{e^{Q_t(a)/\tau}}{\sum_{b} e^{Q_t(b)/\tau} } 
\end{align*}

Try to do it at home!

Softmax Action Selection policy

As the estimates converge, what does the softmax policy become?

\begin{align*}
\pi_t(a) \longrightarrow \frac{e^{q_*(a)/\tau}}{\sum_{b} e^{q_*(b)/\tau}}
\end{align*}

a fixed stochastic policy

τ

P(best action)

expected reward

1

0.32

0.81

0.25

0.74

1.45

0.1

0.89

1.53

10-armed testbed, true values drawn from N(0, 1); optimal expected reward 1.54

With a fixed , the policy converges to softmax(): it learns a policy, but not the optimal
one.

\begin{align*}
\pi_t(a) = \begin{cases} 1 \;\;\text{if}\;\; a = \displaystyle\operatorname*{arg\,max}_{a\in\mathcal{A}}\;\left[Q_t(a) + c\sqrt{\tfrac{\ln(t)}{n_t(a)}}\,\right] \\[1ex] 0 \qquad\quad \text{otherwise} \end{cases}
\end{align*}
\begin{align*}
\pi_t(a) = \begin{cases} (1-\epsilon) + \epsilon/|\mathcal{A}| & \text{if }\; a = \displaystyle\operatorname*{arg\,max}_{a\in\mathcal{A}} Q_t(a) \\ \epsilon/|\mathcal{A}| & \qquad\quad\text{otherwise} \end{cases}
\end{align*}

Algorithms: Action-value methods

Greedy

-greedy

UCB

Softmax

\begin{align*}
 \pi_t(a) =
    \begin{cases}
      1 \hspace{2mm} \text{if} \hspace{2mm} a=\underset{a \in \mathcal{A}}{\arg \max} \; Q_t(a)\\
       0 \hspace{7mm} \text{otherwise}
    \end{cases} 
\end{align*}
\begin{align*}
 \pi_t(a) = \frac{e^{Q_t(a)/\tau}}{\sum_{b} e^{Q_t(b)/\tau} } 
\end{align*}

Algorithms: Policy gradient based

Action
Preferences

Policy

Action Preference update rule

Algorithm

?

?

Policy gradient methods

Goal: Maximize cumulative reward

\begin{align*}
\sum_{i=1}^{t} R_i
\end{align*}

To achieve the goal, we learn a policy : a distribution on

\begin{align*}
A_t \in \mathcal{A}
\end{align*}

Multi-armed bandit problem

Instead of learning first and then deriving , why not learn directly?

Policy gradient methods

Idea: update policy parameters such that expected value increases

​

Use a policy parameterized by :

\begin{align*}
\mathbb{E}[R_t | \pi_\theta]
\end{align*}

Use gradient ascent to update

\begin{align*}
\theta \leftarrow \theta + \alpha \nabla_\theta \mathbb{E}[R_t | \pi_\theta]
\end{align*}
\begin{align*}
\pi_\theta
\end{align*}

Policy gradient methods

Idea: update policy parameters such that expected value increases

​

Use a policy parameterized by :

\begin{align*}
\mathbb{E}[R_t | \pi_\theta]
\end{align*}

Use gradient ascent to update

\begin{align*}
\theta \leftarrow \theta + \alpha \nabla_\theta \mathbb{E}[R_t | \pi_\theta]
\end{align*}
\begin{align*}
\pi_\theta
\end{align*}

Step-size between 0 and 1

“How fast is updated”

Policy gradient methods

Idea: update policy parameters such that expected value increases

​

Use a policy parameterized by :

\begin{align*}
\mathbb{E}[R_t | \pi_\theta]
\end{align*}

Use gradient ascent to update

\begin{align*}
\theta \leftarrow \theta + \alpha \nabla_\theta \mathbb{E}[R_t | \pi_\theta]
\end{align*}

How to compute this gradient?

\begin{align*}
\pi_\theta
\end{align*}

Policy gradient methods

\begin{align*}
\nabla_\theta \mathbb{E}[R_t | \pi_\theta] = \nabla_\theta \sum_a \pi_\theta(a) \mathbb{E}[R_t | A_t = a]
\end{align*}
\begin{align*}
q(a)
\end{align*}

Definition of conditional
expectation on our policy.

\begin{align*}
\pi(a) = \Pr\,\{A_t = a\}, \quad \text{for all } a \in \mathcal{A}
\end{align*}

Policy gradient methods

\begin{align*}
\nabla_\theta \mathbb{E}[R_t | \pi_\theta] = \nabla_\theta \sum_a \pi_\theta(a) \mathbb{E}[R_t | A_t = a]
\end{align*}
\begin{align*}
= \sum_a q(a) \nabla_\theta \pi_\theta(a)
\end{align*}
\begin{align*}
q(a)
\end{align*}

Policy gradient methods

\begin{align*}
\nabla_\theta \mathbb{E}[R_t | \pi_\theta] = \nabla_\theta \sum_a \pi_\theta(a) \mathbb{E}[R_t | A_t = a]
\end{align*}
\begin{align*}
= \sum_a q(a) \nabla_\theta \pi_\theta(a)
\end{align*}
\begin{align*}
= \sum_a q(a) \frac{\pi_\theta(a)}{\pi_\theta(a)} \nabla_\theta \pi_\theta(a)
\end{align*}

Policy gradient methods

\begin{align*}
\nabla_\theta \mathbb{E}[R_t | \pi_\theta] = \nabla_\theta \sum_a \pi_\theta(a) \mathbb{E}[R_t | A_t = a]
\end{align*}
\begin{align*}
= \sum_a q(a) \nabla_\theta \pi_\theta(a)
\end{align*}
\begin{align*}
= \sum_a q(a) \frac{\pi_\theta(a)}{\pi_\theta(a)} \nabla_\theta \pi_\theta(a)
\end{align*}
\begin{align*}
= \sum_a \pi_\theta(a) q(a) \frac{\nabla_\theta \pi_\theta(a)}{\pi_\theta(a)}
\end{align*}
\begin{align*}
= \mathbb{E}\Big[ R_t \frac{\nabla_\theta \pi_\theta(A_t)}{\pi_\theta(A_t)} \Big]
\end{align*}
\begin{align*}
\frac{d}{dx} \log f(x) = \frac{1}{f(x)} \frac{df(x)}{dx}
\end{align*}

Policy gradient methods

\begin{align*}
\nabla_\theta \mathbb{E}[R_t | \pi_\theta] = \nabla_\theta \sum_a \pi_\theta(a) \mathbb{E}[R_t | A_t = a]
\end{align*}
\begin{align*}
= \sum_a q(a) \nabla_\theta \pi_\theta(a)
\end{align*}
\begin{align*}
= \sum_a q(a) \frac{\pi_\theta(a)}{\pi_\theta(a)} \nabla_\theta \pi_\theta(a)
\end{align*}
\begin{align*}
= \sum_a \pi_\theta(a) q(a) \frac{\nabla_\theta \pi_\theta(a)}{\pi_\theta(a)}
\end{align*}
\begin{align*}
= \mathbb{E}\Big[ R_t \frac{\nabla_\theta \pi_\theta(A_t)}{\pi_\theta(A_t)} \Big]
\end{align*}
\begin{align*}
= \mathbb{E}[R_t \nabla_\theta \log \pi_\theta(A_t)]
\end{align*}
\begin{align*}
\frac{d}{dx} \log f(x) = \frac{1}{f(x)} \frac{df(x)}{dx}
\end{align*}

Policy gradient methods

\begin{align*}
\theta \leftarrow \theta + \alpha \nabla_\theta \mathbb{E}[R_t | \pi_\theta]
\end{align*}

Gradient ascent

\begin{align*}
\nabla_\theta \mathbb{E}[R_t | \pi_\theta] = \mathbb{E}[R_t \nabla_\theta \log \pi_\theta(A_t)]
\end{align*}

From previous derivation.

We can sample this and use stochastic gradient ascent on the (true) value of the policy.

\begin{align*}
\theta \leftarrow \theta + \alpha R_t \nabla_\theta \log \pi_\theta(A_t)
\end{align*}

We can use sampled rewards! It does not need value estimates.

Action preferences

Consider a Softmax policy based on action preferences

\begin{align*}
\pi_t(a) = \frac{e^{\textcolor{red}{H_t(a)}}}{\sum_{b} e^{\textcolor{red}{H_t(b)}}}
\end{align*}

The preferences are learnable policy parameters.

Goal: learn by optimizing the preferences with stochastic gradient ascent.

\begin{align*}
\theta \leftarrow \theta + \alpha R_t \nabla_\theta \log \pi_\theta(A_t)
\end{align*}
\begin{align*}
\pi_t(a) = \frac{e^{\textcolor{red}{Q_t(a)}/\tau}}{\sum_{b} e^{\textcolor{red}{Q_t(b)}/\tau}}
\end{align*}

action values

action preferences

→ , so the action-value policy converges to softmax(). has no fixed target.

Action Preferences

\begin{align*}
\theta \leftarrow \theta + \alpha R_t \nabla_\theta \log \pi_\theta(A_t)
\end{align*}
\begin{align*}
H_{t+1}(a) = H_t(a) + \alpha R_t \frac{\partial \log \pi_t(A_t)}{\partial H_t(a)}
\end{align*}
\begin{align*}
\pi_t(a) = \frac{e^{H_t(a)}}{\sum_b e^{H_t(b)}}
\end{align*}

Softmax policy

Stochastic Gradient ascent

Derivation of the
update rule

\begin{align*}
= H_t(a) + \alpha R_t (\mathbb{I}(a = A_t) - \pi_t(a))
\end{align*}
\begin{align*}
H_{t+1}(a) = \begin{cases} H_t(a) + \alpha R_t (1 - \pi_t(a)) & \quad \text{if}\quad a = A_t \\ H_t(a) - \alpha R_t \pi_t(a) & \quad \text{if}\quad a \neq A_t \end{cases}
\end{align*}

Action Preferences

\begin{align*}
\theta \leftarrow \theta + \alpha R_t \nabla_\theta \log \pi_\theta(A_t)
\end{align*}

➢Preferences for actions with higher rewards increase more (or decrease less), making them more
likely to be selected again.

Update Rule

Stochastic Gradient ascent

\begin{align*}
\pi_t(a) = \frac{e^{H_t(a)}}{\sum_b e^{H_t(b)}}
\end{align*}

Softmax policy

\begin{align*}
H_{t+1}(a) = \begin{cases} H_t(a) + \alpha R_t (1 - \pi_t(a)) & \quad \text{if}\quad a = A_t \\ H_t(a) - \alpha R_t \pi_t(a) & \quad \text{if}\quad a \neq A_t \end{cases}
\end{align*}

Action Preferences with baseline

Update Rule

To mitigate this, we can keep tracking of the average of the
rewards up to but not including time
(with ), and
consider it as a baseline.

Recall that the sampled
reward at each time-step
has high variance.

1) = 2

2) = 1

3) = -1

4) = 3

5) = 0

\begin{align*}
R_t \sim \mathcal{R}_a
\end{align*}
\begin{align*}
H_{t+1}(a) = \begin{cases} H_t(a) + \alpha (R_t - \bar{R}_t)(1 - \pi_t(a)) & \quad \text{if}\quad a = A_t \\ H_t(a) - \alpha (R_t - \bar{R}_t) \pi_t(a) & \quad \text{if}\quad a \neq A_t \end{cases}
\end{align*}

Update Rule with baseline

More details on page 38 of the book.

Action Preferences

Average performance of the gradient bandit algorithm with and without a reward
baseline on the 10-armed bandit testbed.

Action Values vs Action Preferences

10-armed testbed, true values drawn from N(0, 1); average of 2,000 runs

Softmax on : converges to softmax(), a fixed stochastic policy.

Action preferences : no fixed target, so the policy becomes nearly deterministic: on the best
action in about 9 of 10 runs, otherwise mostly on the second best.

\begin{align*}
H_{t+1}(a) = \begin{cases} H_t(a) + \alpha (R_t - \bar{R}_t)(1 - \pi_t(a)) & \quad \text{if}\quad a = A_t \\ H_t(a) - \alpha (R_t - \bar{R}_t) \pi_t(a) & \quad \text{if}\quad a \neq A_t \end{cases}
\end{align*}
\begin{align*}
H_{t+1}(a) = \begin{cases} H_t(a) + \alpha R_t (1 - \pi_t(a)) & \quad \text{if}\quad a = A_t \\ H_t(a) - \alpha R_t \pi_t(a) & \quad \text{if}\quad a \neq A_t \end{cases}
\end{align*}
\begin{align*}
\pi_t(a) = \frac{e^{H_t(a)}}{\sum_b e^{H_t(b)}}
\end{align*}

Algorithms: Policy gradient based

Action
Preferences

Policy

Action Preference update rule

Algorithm

Action Preference update rule with baseline

Summary

Parameter study of various bandit algorithms. Each point is the average reward obtained
over 1000 steps with a particular algorithm at a particular setting of its parameter.

See book’s page 42 for details.

Take home message

•Summary

•The exploration vs exploitation dilemma is in the core of many algorithms.

•We can learn a policy by first learning its action-value estimates

•We can learn a policy directly, via action preferences, for instance.

​

•Further studying:

•Chapter 2 – Reinforcement Learning: An introduction

•Short notes: https://people.stfx.ca/jdelamer/courses/csci-531/lectures/multi-armed-bandit/the-
problem.html

•Video: https://www.youtube.com/watch?v=aQJP3Z2Ho8U

​

​

•Next class:

•Markov Decision Process

​