Select a benchmark from the claim
This edition consolidates five overlapping reports. A benchmark includes tasks, observations, actions, rewards, versions, budgets, and testing. Higher return in a fixed environment alone does not establish visual generalization, robot reliability, or lower compute.
| Family | Useful for | Protocol to fix |
|---|---|---|
| Classic Control/Box2D | Debugging and light control | Environment ID, horizon, reward |
| MuJoCo/DM Control | Continuous control and efficiency | Model version, state/pixel inputs, action repeat |
| ALE/Atari | Visual decisions and exploration | Sticky actions, action set, frame skip, evaluation mode |
| Procgen/MiniGrid/BabyAI | Generated-task generalization | Training/test levels and task conditions |
| Meta-World/robosuite/RLBench | Contact manipulation and multitask learning | Task set, success, controller, demonstrations |
| D4RL/Minari datasets | Fixed-data offline learning | Dataset versions, converted semantics, score references |
| Safety-Gymnasium | Costs and constraints | Cost definition, budget, violations |
| Habitat/CARLA | Navigation/driving transfer | Scenes, sensors, challenge version |
Framework names do not identify complete experiments. Migrated datasets need checks of termination fields and normalization references against their original protocol.
Additional suites and systems
DeepMind Lab, Retro, Crafter, and MineRL cover visual exploration, long-horizon behavior, or historical game protocols. Specify versions and dependencies. Isaac Lab, ManiSkill, MuJoCo Playground, Genesis, HumanoidBench, and MyoSuite offer different embodied interfaces and model assumptions. GPU simulation support alone does not establish lower cost for every task.
Unity ML-Agents, PySC2/SC2LE, and Google Research Football can involve teams or competition; disclose the agent’s controlled units. Hardware requirements should be measured for the chosen configuration rather than inferred from context-free throughput numbers.
| Claim | Measurements |
|---|---|
| Sample efficiency | Fixed-transition performance, shared-interval AUC, threshold budget |
| Generalization | Held-out tasks/scenes/levels and train–test gap |
| Manipulation | Original success, contacts, recovery, actual task time |
| Stability | All seeds, failure frequency, intervals, curves |
| System efficiency | Transitions, updates, wall time, hardware, complete overhead |
| Constraints | Reward, cost, violations, operational definitions |
The common expression requires benchmark-specific reference scores; it is not a universal normalization standard.
A practical sequence
Debug a small task, choose a suite matching the central claim, then examine external-task scope. Preserve packages, versions, maps/models, wrappers, resets, termination, seeds, and budgets.
Related reading: MARL benchmarks, statistics, and protocols.