Exercise R.3.5

Gradients: true or false on six identities

Reinforcement Learning › Exercises › Background review › R.3 Expected value
MediumWritten

Mark as T/F. Explain why.

  • \(\nabla_\theta f(s) \pi_\theta (a) = f(s) \nabla_\theta \pi_\theta (a)\)

  • \(\nabla_s f(s)=f(\nabla_s s)\)

  • \(f(\nabla_x x) = f(1)\)

  • \(\nabla_a (\pi(a)+b(s))=\nabla_a \pi(a)\)

  • \(\nabla_\theta \mathbb{E}[(G-\lvert \lvert \theta \rvert \rvert_2)\log \pi_\theta (a)] = \nabla_\theta \mathbb{E}[-\lvert \lvert \theta \rvert \rvert_2\log \pi_\theta (a)]\)

  • \(\sum_a b(s) \nabla_a \pi(a|s, \theta)=0\), assuming swapping the sum and gradient is allowed