Entropy and the soft target

Maximum-entropy RL adds temperature-weighted policy entropy to return. A common twin-critic SAC target is

y=r+γ(1−d)[min⁡jQθˉj(s′,a′)−αlog⁡πϕ(a′∣s′)],a′∼πϕ.y=r+\gamma(1-d)\left[\min_j Q_{\bar\theta_j}(s',a')-\alpha\log\pi_\phi(a'\mid s')\right], \qquad a'\sim\pi_\phi.

The critic fits this target. The actor minimizes E[αlog⁡πϕ(a∣s)−min⁡jQθj(s,a)]\mathbb E[\alpha\log\pi_\phi(a\mid s)-\min_jQ_{\theta_j}(s,a)]. Taking a minimum mitigates some overestimation; it does not prove unbiased values. SAC variants with an explicit value network should be distinguished from later implementations without it.

Temperature as a dual variable

The constraint E[−log⁡π]≥Htarget\mathbb E[-\log\pi]\geq\mathcal H_{target} introduces nonnegative α\alpha. With a fixed policy,

J(α)=E[−α(log⁡π(a∣s)+Htarget)],∂αJ=H(π)−Htarget.J(\alpha)=\mathbb E[-\alpha(\log\pi(a\mid s)+\mathcal H_{target})],\qquad \partial_\alpha J=\mathcal H(\pi)-\mathcal H_{target}.

Entropy above target gives a positive gradient, so descent reduces temperature; low entropy increases it. A source note reversed this sign interpretation. Continuous differential entropy may be negative, making discrete-entropy intuition inappropriate for negative target entropy.

Setting α=exp⁡η\alpha=\exp\eta enforces positivity. The exact derivative of J(exp⁡η)J(\exp\eta) includes an alpha factor. A common log-temperature surrogate omits it, preserving fixed points but changing update scaling. Document the actual loss, detach boundaries, and target.

General dual optimization

Inner primal optimization and multiplier updates follow a stated Lagrangian convention. “Dual ascent” and “dual descent” cannot be compared without that convention. Strong duality and convex guarantees do not automatically apply to joint neural-policy training.

Corrections retained in this edition

Standard SAC is an online off-policy algorithm; fixed offline data require additional coverage treatment. Untraceable source claims of 40% speedup or 2.3-fold success are not promoted into this edition. A sampleable implicit policy also needs the required density calculations to use the standard entropy objective.

Related reading: reparameterization, sampling, and extrapolation.

Selected references from the source notes