State the claim and test split
This protocol combines seven benchmark reports, statistics notes, and supplementary reproducibility reading. Define the research question and primary measurement before fixing tasks and validation data. Planned experiments remain proposals.
| Record | Minimum content |
|---|---|
| Environment identity | Packages, versions, IDs, maps/models, wrappers |
| Task semantics | Rewards, success, costs, horizon, termination/truncation |
| Inputs/actions | Modalities, privileged inputs, units, normalization, repeats |
| Data budget | Joint transitions, agent steps, demonstrations, offline data, resets |
| Optimization | Updates, batches, architectures, optimizer, total search configurations |
| Policy selection | Final/best-validation/checkpoint rule and data |
| Evaluation | Frozen policy, episodes, seeds, noise, test split |
| Systems | CPU/GPU, parallelism, precision, complete wall time |
| Outputs | Raw seed results, curves, intervals, configuration and code versions |
Collection, optimization, validation, and testing consume different budgets. Test-based checkpoint selection introduces bias into the reported test score.
Define metrics before plotting
On the shared interval , use if that is the chosen convention. Threshold budgets need a threshold, smoothing, persistence rule, and treatment of unreached targets.
Success rates require fixed denominators, timeouts, and contact rules. Separate training return from test return. Establish score references before cross-task normalization, and retain per-task results with robust aggregates.
Reproducible execution
Debug rewards, masks, resets, action bounds, and seeds in a small task. Establish baselines before the main study. Match data and search budgets, isolate component ablations, and retain all runs including failures.
Automatic resets require actual final observations. External truncation in continuing tasks normally retains bootstrap, while finite-horizon tasks follow their own state/termination semantics.
Scope of conclusions
Small seed counts do not justify universal superiority. Hardware, reward, maps, and implementations influence rankings. State which fixed conditions improved which metric and identify untested conditions.
Related reading: statistics, single-agent suites, MARL, and reproducibility evidence.