STUDY NOTEBOOK
RL Study Notebook
52 substantive source notes organized into 34 bilingual articles.
Edited guides consolidate repeated discussions, standardize definitions and equations, and connect classic readings to evaluation. Every article links to full original Markdown. Historical study notes are distinguished from verified paper evidence, derivations, and proposed experiments.
25 topic guides and 9 literature records, backed by 51 distinct source texts. Duplicate copies map to shared topics; empty placeholders are not published as articles.
Source mapping and checksum manifest (.json) →RL Foundations and Probability6
- MDPs, Value Functions, and Bellman Errors
Bellman equations, MSBE, MSPBE, sampled TD errors, and double sampling.
- Probability Measures, Empirical Measures, and RL Occupancy
Probability kernels, trajectory laws, stationary distributions, and discounted occupancy measures.
- Contractions, Fixed Points, and Stochastic Approximation
Separate deterministic Bellman contraction from convergence conditions for noisy recursions.
- On-Policy Sampling, Off-Policy Learning, and Replay
Merge duplicate notes and distinguish behavior policies, target policies, and distribution shift.
- Potential-Based Reward Shaping
Telescoping returns, policy invariance, terminal potentials, and reward-scale caveats.
- Markov States, Partial Observations, and Recurrent Memory
Separate physical state, observation, history, and belief without assuming RNN sufficiency.
Policy Optimization and Action Modeling7
- Policy Gradients, REINFORCE, and Surrogate Objectives
Derive the trajectory gradient and connect baselines, PPO clipping, and KL diagnostics.
- Eligibility Traces and TD(λ)
Forward lambda returns, backward traces, online equivalence conditions, and episode boundaries.
- Behavior Cloning, Action Bins, and Multimodal Actions
Supervised objectives, closed-loop distribution shift, and regression, classification, and residual actions.
- Diffusion Policy: Distribution Modeling and Control Timing
A supplement to P08 on denoising time, action chunks, and the execution loop.
- Offline RL and Value Extrapolation Error
Value estimates outside data support, maximization backups, and error propagation.
- Reparameterization and Pathwise Gradients
Gaussian reparameterization, tanh density corrections, and the limits of pathwise differentiation.
- Maximum-Entropy RL, SAC, and Dual Temperature Updates
Merge five notes and correct temperature-gradient signs and the off-policy/offline distinction.
Abstraction, Hierarchy, and Adaptation4
- Options, SMDPs, and Intra-Option Learning
Temporal abstraction, duration-dependent discounting, and experience shared across options.
- MDP Homomorphisms and State–Action Abstraction
Reward and transition preservation, policy lifting, bisimulation, and approximate representations.
- Goal-Conditioned RL, UVFA, and HER
Goal-conditioned values, hindsight relabeling, and hierarchical-control interfaces.
- Meta-RL: Task Distributions and Rapid Adaptation
Optimization, contextual inference, and recurrent adaptation across tasks.
Multi-Agent Learning and Evaluation5
- IPPO, COMA, and Multi-Agent Gradient Variance
Independent learning, centralized critics, counterfactual credit, and a bounded variance example.
- RL Statistics, Confidence Intervals, and Tuning Budgets
Seeds versus episodes, power, bootstrap, IQM, and tuning leakage.
- Single-Agent RL Benchmarks and Metrics
Consolidate five reports across control, visual generalization, offline datasets, and embodied tasks.
- Multi-Agent RL Benchmarks and Cooperative Evaluation
Cooperative, competitive, mixed-motive, partially observed, and cross-partner MARL evaluation.
- A Reproducible RL Evaluation and Reporting Protocol
Align environment versions, training budgets, checkpoint selection, held-out testing, and aggregation.
Engineering and Embodied Research3
- Simulators, World Models, and Planning
Experimental environments, learned dynamics, and internal rollout serve different roles.
- Embodied RL, LLMs/VLAs, and Research Design
Turn sample efficiency, adaptation, rewards, and timing into controlled research questions.
- TD7 Policy Checkpoints: Separating Training and Evaluation
Candidate and deployed policies, assessment, and independent evaluation.
Classic Papers, Perspectives, and Textbooks9
- A Closer Look at Deep Policy Gradients
Gradient estimates, value prediction, and surrogate reward landscapes.
- Deep Reinforcement Learning that Matters: Supplementary Evidence
How seeds, architectures, reward scales, and implementations change algorithm comparisons.
- Era of Experience: Notes on Experience-Driven Learning
Separate experience-driven learning as a paradigm from world models as a mechanism.
- Fast Gradient-Descent Methods for Temporal-Difference Learning with Linear Function Approximation
MSPBE and auxiliary variables connect off-policy stability to linear-time updates.
- For SALE: State-Action Representation Learning for Deep Reinforcement Learning
SALE objectives, TD7 engineering components, and the scope of empirical evidence.
- High-Dimensional Continuous Control Using Generalized Advantage Estimation
TD-residual advantages, reward-to-go, lambda returns, and critic error.
- Policy Gradient Methods for Reinforcement Learning with Function Approximation
Occupancy distributions, compatible features, and theoretical assumptions for approximation.
- Potential-Based Shaping and Q-Value Initialization are Equivalent
A Q-value shift explains conditional path equivalence between shaping and initialization.
- Reinforcement Learning: An Introduction — A Reading Route
A chapter route from tabular methods through approximation, exploration, and policy gradients.
Suggested reading routes
- Theory: MDPs/probability → contractions and stochastic approximation → TD/traces → policy gradients/GAE.
- Control: cloning/bins → Diffusion Policy → SAC/representations → goals, options, and Meta-RL.
- Experiments: benchmark selection → statistics/tuning → reproducible protocols → embodied research questions.