Reinforcement Learning
An introductory course in eight lectures, with video recordings, tutorials and exercises, following Sutton and Barto's textbook.
The course goes from the exploration–exploitation dilemma in multi-armed bandits, through finite Markov decision processes, dynamic programming, and Monte Carlo and temporal-difference learning, to function approximation, policy gradients and model-based methods. It follows R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, 2nd ed., 2018, free online from the authors.
Course map
Each concept of the lectures, in the order the slides teach it. The slide link opens the deck at the first slide on the concept; the task links open the tutorial at the task that practises it, and the exercise links open the exercises on the concept, numbered lecture.concept.n.
EasyMediumHardCodehas a coding partEx 1.9.3dashed: filed under another concept, also practises this one
R Background review 8 topics · probability and functional analysis
- TopicSlidesWhat it coversExercises
- R.1Random variables L1 · 8–10 Discrete and continuous random variables, PMF and PDF Ex R.1.1Ex R.1.2Ex R.1.3Ex R.1.4Ex R.1.5Ex R.1.6Ex R.1.7Ex R.1.8Ex R.1.9Ex R.1.10Ex R.1.11Ex R.4.4
- R.2Sampling from a distribution L1 · 12 Sampling from a probability distribution
- R.3Expected value L1 · 14–17 Expected value vs sample mean (dice, coin) Ex R.3.1Ex R.3.2Ex R.3.3Ex R.3.4Ex R.3.5Ex R.4.1Ex R.4.2Ex R.4.3Ex 2.4.4Ex 2.4.5Ex 2.4.6Ex 2.4.7Ex 5.5.4
- R.4Probability mass functions L1 · 26 Conditions for a valid PMF Ex R.4.1Ex R.4.2Ex R.4.3Ex R.4.4Ex R.4.5Ex R.4.6Ex 2.4.4
- R.5Step-size conditions L1 · 38 Why the step size guarantees learning: series that diverge and converge Ex R.5.1Ex R.5.2Ex R.5.3Ex R.5.4Ex R.5.5Ex R.5.6Ex R.5.7Ex 4.1.2Ex 5.2.2Ex 5.2.3
- R.6Operators L3 · 47–52 Operator; the Bellman operators as an example
- R.7Fixed points L3 · 53–55 Fixed point of an operator, with an example Ex 3.6.1Ex 3.6.3
- R.8γ-contraction L3 · 56–58 γ-contraction mapping on a normed vector space Ex 3.6.1Ex 3.6.2Ex 3.6.3
1 Exploration Methods Slides (102) Video Tutorial 1 9 concepts
- ConceptSlidesTutorial tasksExercises
- 1.1Multi-armed bandit problem Slides 6–30 T1 Task 1T1 Task 2
- 1.2Sample-average method Slides 33–38 T1 Task 4T1 Task 6 Ex 1.2.1Ex 1.5.5Ex 1.9.3
- 1.3Weighted-average method (nonstationary problems) Slides 39–46 T1 Task 5T1 Task 7 Ex 1.3.1Ex 1.3.2Ex 1.3.3Ex 1.3.4Ex 1.9.3
- 1.4Greedy Slides 49–53 T1 Task 8T1 Task 10 Ex 1.4.1Ex 1.4.2Ex 1.5.5Ex 1.9.2Ex 1.9.3Ex 5.2.2
- 1.5ε-greedy Slides 54–63 T1 Task 12T1 Task 13T1 Task 14 Ex 1.5.1Ex 1.5.2Ex 1.5.3Ex 1.5.4Ex 1.5.5Ex R.4.6Ex 1.9.2Ex 1.9.3Ex 5.2.3Ex 5.5.3Ex 5.6.1
- 1.6Optimistic initial values Slides 64–69
- 1.7Upper confidence bound (UCB) Slides 71–77 Ex 1.7.1Ex 1.9.2Ex 1.9.3
- 1.8Softmax action selection Slides 79–81 Ex 1.9.2Ex 1.9.3Ex 1.9.4
- 1.9Policy gradient methods: action preferences Slides 83–100 Ex 1.9.1Ex 1.9.2Ex 1.9.3Ex 1.9.4
2 The RL Problem and Finite MDPs Slides (77) Video Tutorial 2 6 concepts
- ConceptSlidesTutorial tasksExercises
- 2.1Markov decision process Slides 15–26 T2 Case 2, Task 4 Ex 2.1.1Ex 2.2.1Ex 2.2.2
- 2.2Markov property Slides 27–29 T3 Case 5, Task 4 Ex 2.2.1Ex 2.2.2Ex 2.4.5Ex 2.4.6Ex 2.4.7
- 2.3Returns and discounting Slides 34–40 T2 Case 1, Task 2T2 Case 1, Task 3 Ex 2.3.1Ex 2.3.2Ex 2.3.3Ex 2.3.4Ex 2.4.5Ex 2.4.6Ex 2.4.7Ex 2.5.1Ex 3.6.2Ex 3.6.3Ex 4.1.1Ex 4.2.1Ex 5.2.2
- 2.4Bellman expectation equations Slides 47–61 T3 Case 3, Task 5 Ex 2.4.1Ex 2.4.2Ex 2.4.3Ex 2.4.4Ex 2.4.5Ex 2.4.6Ex 2.4.7Ex 2.6.2Ex 3.1.1Ex 3.1.2Ex 3.3.1Ex 7.5.3
- 2.5Bellman optimality equations Slides 62–69 Ex 2.5.1Ex 3.4.1Ex 3.4.2
- 2.6Bellman equation in matrix form Slides 73–74 T3 Case 3, Task 6T3 Case 3, Task 7 Ex 2.6.1Ex 2.6.2Ex 3.4.3
3 Dynamic Programming Slides (78) Video Tutorial 3 7 concepts
- ConceptSlidesTutorial tasksExercises
- 3.1Policy evaluation Slides 12–19 T3 Case 3, Task 7 (direct solve) Ex 3.1.1Ex 3.1.2Ex 3.3.1Ex 3.3.3Ex 3.4.3Ex 3.4.4Ex 5.3.5
- 3.2Policy improvement theorem Slides 20–36 T3 Case 4, Task 3T3 Case 4, Task 5 Ex 3.3.4
- 3.3Policy iteration Slides 37–39 Ex 3.3.1Ex 3.3.2Ex 3.3.3Ex 3.3.4Ex 3.4.3Ex 3.4.4
- 3.4Value iteration Slides 40–44 Ex 3.4.1Ex 3.4.2Ex 3.4.3Ex 3.4.4Ex 3.6.1Ex 3.6.2Ex 3.6.3Ex 5.3.5
- 3.5Generalized policy iteration (GPI) Slides 45
- 3.6Bellman operators and the Banach fixed-point theorem Slides 46–66 Ex 3.6.1Ex 3.6.2Ex 3.6.3
- 3.7Synchronous and asynchronous DP Slides 67–76
4 Monte Carlo Methods and TD Learning Slides (68) Video Tutorial 4 5 concepts
- ConceptSlidesTutorial tasksExercises
- 4.1Monte Carlo prediction Slides 7–17 T4 Case 1, batch MCT4 Case 1, constant-α MCT3 Case 5, Task 1 Ex 4.1.1Ex 4.1.2Ex 4.1.3Ex 4.3.2Ex 4.5.2Ex 6.1.2
- 4.2Temporal-difference learning: TD(0) Slides 18–29 T4 Case 2, TD(0) Ex 4.2.1Ex 4.2.2Ex 4.3.1Ex 4.3.2Ex 4.3.3Ex 4.3.4Ex 4.5.2Ex 5.2.2Ex 5.3.5Ex 5.7.2Ex 6.1.2Ex 7.5.1Ex 7.5.2Ex 7.5.3
- 4.3MC vs TD: bias and variance, batch updates Slides 30–39 T4 Case 2, The Long JourneyT4 Case 2, final task Ex 4.3.1Ex 4.3.2Ex 4.3.3Ex 4.3.4Ex 8.1.1
- 4.4Multi-step TD Slides 40–45 Ex 4.4.1Ex 8.1.2
- 4.5Mixed multi-step returns and eligibility traces Slides 46–66 Ex 4.5.1Ex 4.5.2
5 Model-Free Control Slides (62) Video Tutorial 5 7 concepts
- ConceptSlidesTutorial tasksExercises
- 5.1Monte Carlo control, GLIE Slides 14–21 Ex 5.2.1Ex 5.2.2Ex 5.2.3
- 5.2SARSA: on-policy TD control Slides 22–24 Ex 5.2.1Ex 5.2.2Ex 5.2.3Ex 5.3.1Ex 5.3.2Ex 5.3.4Ex 5.3.5Ex 5.3.6Ex 5.6.1Ex 6.3.1Ex 6.3.2
- 5.3Q-learning (off-policy learning) Slides 26–30 T5 Case 2, Build the Q-learning agentT5 Case 2, Evaluation protocol Ex 5.3.1Ex 5.3.2Ex 5.3.3Ex 5.3.4Ex 5.3.5Ex 5.3.6Ex 5.5.1Ex 5.5.2Ex 5.5.4Ex 5.6.1Ex 5.7.1Ex 6.3.2Ex 8.1.2
- 5.4Expected SARSA Slides 32 Ex 5.5.3
- 5.5Maximization bias and double Q-learning Slides 34–42 Ex 5.5.1Ex 5.5.2Ex 5.5.3Ex 5.5.4Ex 5.6.1
- 5.6Deep Q-networks: experience replay, target network Slides 44–48 T6 Train your first DQN agentT6 DQN hyperparameters Ex 5.6.1Ex 6.4.1Ex 6.4.2Ex 6.4.4
- 5.7Importance sampling Slides 50–60 Ex 5.7.1Ex 5.7.2Ex 7.2.1
6 Function Approximation Slides (65) Video Tutorial 6 4 concepts
- ConceptSlidesTutorial tasksExercises
- 6.1Value function approximation by stochastic gradient descent Slides 14–35 Ex 6.1.1Ex 6.1.2
- 6.2Linear function approximation Slides 40–51 Ex 6.2.1Ex 6.2.2Ex 6.3.1Ex 6.4.3
- 6.3Control with value-function approximation Slides 52–57 Ex 6.3.1Ex 6.3.2Ex 6.4.5
- 6.4Convergence, divergence and experience replay Slides 58–63 Ex 6.4.1Ex 6.4.2Ex 6.4.3Ex 6.4.4Ex 6.4.5
7 Policy Gradients and Actor-Critic Slides (58) Video Tutorial 7 5 concepts
- ConceptSlidesTutorial tasksExercises
- 7.1Policy-based RL: parameterized policies Slides 9–18 T7 Build the policy networkT7 Gaussian policy Ex 7.1.1Ex 7.1.2Ex 7.1.3Ex 7.1.4Ex 7.1.5Ex 7.1.6Ex 7.4.2
- 7.2Policy-gradient theorem Slides 19–30 T7 Task 2, REINFORCE loss Ex 7.2.1Ex 7.3.1Ex 7.4.3
- 7.3REINFORCE Slides 31–32 T7 Task 3T7 Full training run Ex 7.3.1Ex 7.4.1Ex 7.4.2Ex 7.4.4Ex 7.5.4
- 7.4Reducing the variance: causality and baseline Slides 33–40 Ex 7.4.1Ex 7.4.2Ex 7.4.3Ex 7.4.4Ex 7.5.1Ex 7.5.2Ex 7.5.4
- 7.5Actor-critic, advantage function, A2C Slides 41–54 Ex 7.5.1Ex 7.5.2Ex 7.5.3Ex 7.5.4
8 Model-Based Reinforcement Learning Slides (45) Video 2 concepts
- ConceptSlidesTutorial tasksExercises
- 8.1Model-based RL and Dyna-Q Slides 11–30 Ex 8.1.1Ex 8.1.2Ex 8.1.3
- 8.2Planning for action selection: Monte Carlo tree search Slides 31–42
© 2025–2026 Rafael Fernandes Cunha · Originally developed for Reinforcement Learning (WBAI061-05), University of Groningen.