Scope of this reading route
Sutton and Barto’s Reinforcement Learning: An Introduction, second edition (2018), builds a common language from tabular learning to approximation and policy gradients. This is an edited chapter route from the upload, not a new result or a full textbook translation.
| Stage | Topics | Questions |
|---|---|---|
| Tabular methods | Bandits, MDPs, DP, Monte Carlo, TD, n-step, planning | Expectations/samples, evaluation/control, exploration |
| Approximation | On/off-policy prediction/control, traces, policy gradients | Distribution, projection, bootstrap, gradients, stability |
| Deeper connections | Psychology/neuroscience, applications, frontiers | Distance between assumptions and systems |
Prerequisite sequence
Start with probability and values, then compare exact DP, sampled Monte Carlo, and bootstrapped TD. Continue through n-step learning and traces.
For approximation, specify features, fitting distribution, and operators together. GTD2/TDC explains why randomizing replay does not alone ensure stability. Connect policy gradients, compatible approximation, and GAE.
Small, checkable tasks
Use bandits for exploration, grid MDPs for iteration, random walks for MC/TD/n-step, and short chains for forward/backward traces. Shared data and computable values separate update objectives from superficial curves.
Before deep continuous control, add benchmark selection and statistics. Neither textbook theory nor a benchmark score alone establishes arbitrary neural implementation correctness.
Keeping notes
Record definitions, assumptions, derivations, counterexamples, and one checkable example per chapter. Separate open questions from proved statements. The heading-only GAE placeholder is not published; substantive GAE material appears in P36.