Exercise R.4.1

Expected value: gradient of E[log π(X)] is zero

Reinforcement Learning › Exercises › Background review › R.4 Probability mass functions
EasyWritten

Show that for a discrete random variable \(X\) with a pmf \(\pi(x)\), \(\nabla_x \mathbb{E}(\log \pi(X)) = 0\).

Hint: Here swapping the gradient and a sum is correct (if interested, check uniform convergence and derivative of log-likelihood). It is generally allowed to swap the gradient and sum whenever we are working with sums of pmfs, expected values, or sample means (whenever they are finite).

Hint: Push the gradient in, use the definition of expected value, use the properties of the derivative of the natural logarithm, then swap the gradient and the sum, the pmf must sum to 1, the gradient of 1 is 0.