Identify the statistical unit
One training seed represents one training run. More evaluation episodes of the same model reduce evaluation noise, not training variability. Transitions, checkpoints, and fragments from the same run are not independent algorithm replications. This determines the level at which error bars and resampling operate.
For independent seed-level outcomes,
Standard deviation describes dispersion; standard error describes mean-estimation precision. A confidence interval needs its method, statistic, and sampling unit.
Power and multiple comparisons
Significance level controls type-I error; power depends on effect size, variance, sample size, test, and design. Five seeds are not a universal adequacy rule. Pilot data can inform a budget for a meaningful difference. Report effect sizes and uncertainty alongside significance.
Multiple methods, tasks, and checkpoints create selection issues. Failure to reject a null is not proof of equivalence; equivalence and noninferiority require their own margins and design.
Aggregation across tasks
Fix reference scores before normalization. IQM averages the middle 50% of scores and reduces extreme-tail influence without removing uncertainty. Stratified bootstrap should reflect the relevant task/seed hierarchy. Generalization to new tasks additionally requires stating whether tasks themselves are sampled.
Performance profiles, threshold budgets, improvement probabilities, and per-task results complement aggregates. Learning-curve AUC needs a shared interval and integration convention.
Tuning is a budget
Grid search evaluates a specified configuration set. Report its size, seeds per configuration, selection data, and total compute. Logarithmic scales can be more appropriate for parameters spanning orders of magnitude. Repeated test-set selection leaks information. The empirical-measure material is connected to the probability guide.
Related reading: reporting protocols and Deep RL That Matters.