State what should be unfamiliar
An evaluation can test new attempts of a familiar setup, new objects, new tasks, new environments, or a different robot. These are distinct questions. Choose the intended boundary before splitting records. A random episode split may be suitable for one question and inadequate for another, particularly when repeated attempts share nearly identical scenes and instructions.
Group related data at the source
Keep overlapping windows, transformed copies, and repeated exports attached to their original episode. Consider grouping sessions, setups, or operators when relevant to the design. Preserve these references through processing. Different filenames and encoding settings do not establish independence, and a common task label does not reveal whether two examples came from the same physical attempt. Link resets and retries to their shared collection context.
Check coverage in each partition
Review tasks, outcomes, objects, viewpoints, and embodiments after applying the split. Some groups may contain only one source or control condition, limiting interpretation. Report those limitations rather than silently relaxing the boundary to improve balance. If failure examples are scarce, keep that constraint visible when describing what the evaluation can meaningfully measure.
Version and maintain the assignment
Store the grouping rule, randomization details where used, and exact episode assignments. New deliveries and revised clips should follow the same source-level logic. Recheck overlap when the collection grows or mappings change. A folder named evaluation is only an organizational label; the evidence for independence lies in the documented grouping and source relationships.
Review checklist
- Name the intended generalization boundary
- Group derived records with their sources
- Review coverage after splitting
- Preserve versioned partition assignments