An evaluation set should distinguish learning performance from data integrity and task-definition problems. State what the robot must accomplish, which conditions change, and how outcomes are assessed. Sourcing discussions can then focus on the evidence and variation needed to answer that question, with project-specific review of availability and proposed use.
Specify the evaluation unit
Decide whether an example is a complete attempt, a transition, a recovery, or an instruction-linked episode. Record initial conditions and resets. If a task is interrupted or assistance occurs, retain that information rather than presenting the remaining movement as an ordinary independent completion.
Define outcome evidence
Success can depend on placement, object state, or another task-specific result. Explain how the label is established and what happens when the evidence is missing. Keep procedural correctness, final outcome, and termination reason separate. A stopped episode does not necessarily mean that the intended task was completed.
Choose variation deliberately
Identify held-out objects, layouts, tasks, or embodiments and preserve grouping information needed to maintain those boundaries. Review whether challenging conditions appear only with one source or operator. Such associations can complicate interpretation even when every file and annotation is technically valid.
Version the benchmark and its review
Retain manifests, field definitions, exclusions, and correction history. Validate sensor availability and action meaning before measuring a model. For uncertain outcomes, establish a review procedure and keep the uncertainty visible. Evaluation results should be interpreted against the exact dataset version and conditions, not a broad category label. Identify who resolves disputed outcomes and retain their evidence references. Report exclusions by reason so later benchmark revisions can distinguish corrected records from a changed evaluation scope.
Include in your inquiry
- Episode unit and reset conditions
- Outcome and termination definitions
- Held-out condition groups
- Integrity checks and benchmark version