Scope: supplementary evidence
The uploaded note concentrates on the supplemental material of Deep Reinforcement Learning that Matters: literature configurations, full settings, architecture/activation/reward-scale studies, code comparisons, and statistics. One reported configuration is not a universal optimum.
The source links arXiv:1709.06560, a 2017 preprint associated with AAAI 2018. Reported results depend on historical environment versions and should not be compared directly with current environments sharing a name.
Algorithm names do not define baselines
DDPG, TRPO, PPO, and ACKTR implementations can differ in architecture, activation, reward scale, batch size, and evaluation. The notes list historical network examples such as , , and ; these illustrate experimental variables rather than recommended common defaults.
Critic choices influence actor estimates and updates. Disclose implementation differences and use comparable search/optimization budgets.
Reward and batch effects
The notes describe strong DDPG reward-scale sensitivity and historical TRPO settings where larger batches did not help. Targets, gradient scales, update frequency, and stale policy data can all matter. These observations are configuration-specific, not universal batch-size laws.
Seeds, implementations, and selection
Between-run or between-codebase variation can rival a claimed improvement. Reporting only top seeds or the best tested checkpoint changes the estimand. Evaluation also needs complete episodes under a specified frozen policy rather than an unclear mixture of changing policies.
A reporting protocol
Preserve independent runs and distinguish training, assessment, and test. Publish environments, wrappers, networks, rewards, optimizers, batches, search budgets, and interval methods. Interpret significance alongside practical effect size and select sample counts from variance and detectable differences.
Related reading: statistics, protocols, and benchmarks.