Exercise 1.9.2
Exploration methods: deterministic versus stochastic policies, hyperparameters, baseline
Answer the following theoretical questions about exploration methods.
1. What is the main difference between a deterministic policy and a stochastic policy?
2. How is exploration driven in a stochastic policy?
3. Classify the policies greedy, \(\epsilon\)-greedy, UCB, Softmax, and Action Preference as stochastic or deterministic.
-
4. Explain the function of the hyperparameters below. What effect does a high and low hyperparameter value have for each of these policies?
a) \(\epsilon\) in \(\epsilon\)-greedy.
b) \(c\) in UCB.
c) \(\tau\) in Softmax.
d) \(\alpha\) in action preference.
5. Why does the use of a baseline for the Action Preference policy improve performance?