Preserve what was actually requested
An instruction given before execution can differ from a later description of what occurred. Keep those fields separate and record when instructions change. If an operator corrects a target halfway through a task, preserve the timing and relationship to the episode. Rewriting the original instruction to match the final action can hide meaningful errors and interventions.
Resolve references in the scene
A phrase such as the small box or the left container needs enough context to identify the intended object. Define whether labels use object instances, categories, spatial relations, or other references. Include examples with similar distractors. If the instruction cannot be resolved uniquely from the provided observation, mark the ambiguity rather than assigning an arbitrary correct target.
Link text to behavior and outcomes
Specify whether text applies to the whole episode, one subtask, or a time interval. Task success should come from the agreed outcome evidence, not from the presence of a matching instruction. A robot can perform a plausible action on the wrong object. Review those distinctions when creating language-conditioned examples and when deciding which fields the learner receives.
Plan language variation deliberately
Paraphrases can vary wording while leaving the physical task unchanged. New objects and goals create different changes. Track these separately so evaluation can distinguish linguistic variation from new task conditions. Keep answer annotations and retrospective summaries out of model inputs when the evaluation requires them to remain hidden. Include examples of all text types in the sourcing brief.
Review checklist
- Separate instructions and summaries
- Define object-reference rules
- Link text to episode intervals
- Track linguistic and physical variation