Deriving the trajectory gradient

For J(θ)=Eτ∼pθ[∑u=0T−1γuru]J(\theta)=\mathbb E_{\tau\sim p_\theta}[\sum_{u=0}^{T-1}\gamma^u r_u], assume environment transitions have no direct dependence on policy parameters. Differentiating trajectory log probability leaves policy score terms. Past rewards can be removed by conditional independence, giving

∇J(θ)=E ⁣[∑t=0T−1γt∇θlog⁡πθ(at∣st)Gt],Gt=∑l=0T−t−1γlrt+l.\nabla J(\theta)=\mathbb E\!\left[\sum_{t=0}^{T-1}\gamma^t\nabla_\theta\log\pi_\theta(a_t\mid s_t)G_t\right],\qquad G_t=\sum_{l=0}^{T-t-1}\gamma^l r_{t+l}.

The outer γt\gamma^t is explicit. Other conventions absorb it into discounted occupancy; silently removing it while retaining the same objective changes the derivation.

Baselines and variance

For a fixed state, Ea[∇log⁡π(a∣s)b(s)]=0\mathbb E_a[\nabla\log\pi(a\mid s)b(s)]=0 when the baseline is action-independent. A value baseline yields Aπ=Qπ−VπA^\pi=Q^\pi-V^\pi, although it is not necessarily optimal for every score-weighted variance criterion.

REINFORCE noise comes from long stochastic returns, policy scores, sparse success, and other agents. Reward-to-go removes irrelevant past rewards. Critics, GAE, and normalization further affect updates, with approximation error and finite-sample effects. Low value loss alone does not establish a good actor gradient.

Surrogates and PPO

On fixed old-policy data, a common surrogate is L(θ)=Eπold[ρtA^t]L(\theta)=\mathbb E_{\pi_{old}}[\rho_t\hat A_t], with ρt=πθ/πold\rho_t=\pi_\theta/\pi_{old}. PPO uses

Lclip=E ⁣[min⁡(ρtA^t,clip⁡(ρt,1−ϵ,1+ϵ)A^t)].L^{clip}=\mathbb E\!\left[\min\bigl(\rho_t\hat A_t,\operatorname{clip}(\rho_t,1-\epsilon,1+\epsilon)\hat A_t\bigr)\right].

Clipping limits beneficial surrogate incentives in selected directions. It is not a hard KL constraint or a guarantee of monotonic true-return improvement. Repeated optimization can make advantages and the old state distribution increasingly stale. Monitor KL, clip fraction, entropy, gradient scale, and independent evaluation return.

Connecting theory to measurements

Evaluate gradient accuracy, true-value prediction, and surrogate/return alignment separately. A method can improve one while worsening another.

Related reading: the original theorem, GAE, A Closer Look, and multi-agent gradients.

Selected references from the source notes