Exercise R.3.5
Gradients: true or false on six identities
Mark as T/F. Explain why.
\(\nabla_\theta f(s) \pi_\theta (a) = f(s) \nabla_\theta \pi_\theta (a)\)
\(\nabla_s f(s)=f(\nabla_s s)\)
\(f(\nabla_x x) = f(1)\)
\(\nabla_a (\pi(a)+b(s))=\nabla_a \pi(a)\)
\(\nabla_\theta \mathbb{E}[(G-\lvert \lvert \theta \rvert \rvert_2)\log \pi_\theta (a)] = \nabla_\theta \mathbb{E}[-\lvert \lvert \theta \rvert \rvert_2\log \pi_\theta (a)]\)
\(\sum_a b(s) \nabla_a \pi(a|s, \theta)=0\), assuming swapping the sum and gradient is allowed