The theoretical question

Sutton, McAllester, Singh, and Mansour’s NeurIPS 1999 paper connects stochastic policy gradients to function approximation. It avoids explicitly differentiating state visitation and identifies conditions under which a compatible value approximator preserves the gradient expectation.

Objective and occupancy conventions

For fixed initial distribution and normalized discounted occupancy,

∇θJ(θ)=11−γEs∼dπ,a∼πθ[∇θlog⁡πθ(a∣s)Qπ(s,a)].\nabla_\theta J(\theta)=\frac1{1-\gamma}\mathbb E_{s\sim d^\pi,a\sim\pi_\theta}\left[\nabla_\theta\log\pi_\theta(a\mid s)Q^\pi(s,a)\right].

Unnormalized occupancy absorbs the leading factor. Average-reward formulations need their own stationary-distribution and differential-value assumptions; they cannot be silently mixed with discounted identities.

Compatible approximation

Use score features ψθ=∇θlog⁡πθ\psi_\theta=\nabla_\theta\log\pi_\theta and fw=w⊤ψθf_w=w^\top\psi_\theta. Least-squares fitting under the theorem’s matching weights and stationarity conditions yields the relevant residual orthogonality:

E[ψθ(s,a)(Qπ(s,a)−fw(s,a))]=0.\mathbb E\left[\psi_\theta(s,a)\bigl(Q^\pi(s,a)-f_w(s,a)\bigr)\right]=0.

A state baseline can be handled separately. Compatibility is not simply shared actor/critic neural layers, and it does not make every approximate value estimator unbiased.

Natural-gradient connection

Under corresponding weighting, F=E[ψψ⊤]F=\mathbb E[\psi\psi^\top] relates to Fisher geometry, and fitted ww connects to F−1gF^{-1}g. Invertibility, regularization, distribution, and optimization accuracy must be stated. Natural and ordinary gradients differ geometrically, beyond a generic stability claim.

Reading route

Begin with trajectory likelihood, align weights through occupancy, then read GAE and empirical scrutiny. The theorem gives a conditional exact relation; deep implementations need checks that its conditions hold.

Selected references from the source notes