01 / MOTIVATION
Introduction
Imagine asking a person to operate a robot using only a task instruction and live camera views. They can see the objects and control the robot, but they do not know the exact target positions or tolerances written into the benchmark’s success checker. Does the instruction give them enough information to understand what counts as success?
If the person fails to manipulate the robot, that alone tells us little about the instruction. The more revealing case is when their behavior reasonably fulfills the stated request, a human reviewer would consider the task complete, and the benchmark still rejects it. The code may be testing a precise, reproducible condition that the instruction never communicated: a particular compartment, a narrower placement region, or an unstated orientation.
That gap matters for more than evaluation. A demonstration may satisfy the hidden condition while its instruction describes a broader or different goal. The recording teaches a convention that the language does not explain.
Why mismatched supervision matters
Consider an already capable generalist policy with a useful understanding of everyday instructions. Post-training it on these pairs could push it toward benchmark-specific interpretations that conflict with that prior understanding. The model is rewarded for reproducing an unspoken convention, potentially at the expense of behavior that transfers to other settings.
This is our motivation, not a post-training result. We did not update policy weights or measure a loss of generalization. We instead ask a prior question: can we make the instruction and evaluator agree more closely while keeping the expensive recorded demonstrations usable?
Our approach is to use GPT as a proxy for an instruction-following human operator. Its ability to interpret language, inspect images, and interact through a robot harness makes this audit practical now. It is fallible, but its disagreements with a benchmark provide concrete cases to inspect and repair.
02 / METHOD AND BENCHMARKS
Run an operator.
Audit the disagreement.
We built control harnesses for two representative benchmarks: LIBERO, whose task instructions are fixed strings, and RoboTwin, whose instructions vary with the generated scene. GPT-6-Astra operates the robot from task language, live RGB observations, and robot proprioception.
The baseline harnesses follow the task-information boundary commonly used in VLA/WAM evaluation: the policy does not receive expert demonstrations, success-checker code, hidden object poses, rewards, or native success labels. The adapters provide camera calibration and generic control guidance so the agent can use the robot interface; they do not give it a long task-specific solution. This is a common information boundary, rather than a claim that GPT and a VLA/WAM use identical observation or action interfaces.
| Benchmark | Coverage | Instruction layer | Intervention |
|---|---|---|---|
| LIBERO | 40 tasks · 10 fixed initial states each | A fixed sentence for each task | Small edits to nine task instructions |
| RoboTwin | 50 tasks · 10 planned rollouts each | Scene-dependent generated language | Supplement generated instructions with task goal specifications |
We first evaluated the original instructions and compared the agent’s completion judgment with the benchmark’s verdict. Some rejected rollouts appeared to satisfy a reasonable reading of the instruction. We then inspected the videos, task definitions, and demonstration references to identify missing goal information. The audit can use those sources; the acting policy does not receive them directly. The revised instruction is the channel through which the repair reaches the policy.
Read the instruction.
Observe the scene.
GPT controls the robot without demonstration videos or direct access to the success checker.
Looks complete.
Scores as failure.
Review disagreements to separate missing goal information from mistakes by the agent.
Clarify what
completion requires.
Edit LIBERO’s task text or supplement RoboTwin’s instruction-generation chain.
Keep the checker.
Reuse the recordings.
Improve language–goal alignment without changing the native task or recollecting demonstrations.
Two instruction layers: fixed task sentences in LIBERO; scene-dependent generated instructions in RoboTwin.
For LIBERO, we manually reviewed completion claims rejected by the checker in the before-and-after comparison. The review retained the disagreements that followed a reasonable reading of the instruction and rejected one claim: the agent mistook a cabinet for the stove. The LIBERO tables apply that correction; the original claim remains in the review records. The RoboTwin tables use the archived completion rule and should not be read as an equivalent human adjudication of every disagreement.
Instinct-alignment score
We define the instinct-alignment score (IAS) to summarize how often a benchmark rejects an outcome the operator considers complete:
Higher IAS means fewer such disagreements under the stated accounting rule. It is not a replacement for native task success: an agent that never claims completion on a failed rollout can obtain 100% IAS without solving the task. We report the two-by-two judgment tables, native success, and videos together.
Repair the instruction layer
Changing the verifier can redefine which recorded trajectories count as valid demonstrations and force a broader data audit or recollection. We take the less disruptive route: keep the native task and checker fixed, and ask the instruction-refining agent to communicate their goal more clearly. The demonstration recordings remain unchanged; a training pipeline can reuse them with the corrected language rather than collect new trajectories.
Protocol, accounting, and what the policy could see
All stages requested gpt-6-astra with high reasoning. LIBERO used task initial-state IDs and seeds 0–9, 500 control steps, 750 tool calls, and a 3,600-second wall-clock limit. RoboTwin requested ten rollouts for each of 50 official tasks. Neither evaluation involved policy fine-tuning or task-specific cross-episode memory. Native success could terminate the environment automatically.
The final LIBERO revisions use short natural-language edits without exact coordinates or action recipes. RoboTwin’s revised prompts are richer: their public goal specifications can include numeric tolerances and orientation targets derived from the checker. They still do not expose live hidden object state or the checker itself. The two interventions therefore change the instruction layer at different levels and with different amounts of added information.
Under the reporting convention, every native success is counted as complete on the agent side; \ marks the opposite combination as inapplicable. For native failures, retained explicit completion claims count as complete; corrected claims, explicit inability, and missing completion claims count as incomplete. LIBERO has five original and nine final-composite failures without a finish declaration. These assignments are accounting rules, not independent human ratings of every terminal image. See the detailed audit.
03 / INSTRUCTION CHANGES AND RESULTS
Two instruction layers.
The same repair principle.
Keep the physical task, verifier, and demonstration recordings intact. Change how the intended goal is communicated.
3.1 LIBERO: edit the task sentence
The original 400 episodes contain 47 retained completion claims rejected by the checker, giving IAS 88.25%. After the instruction revisions and the documented human correction, that count falls to zero and IAS reaches 100%. Native success rises from 322/400 to 357/400 (80.50% → 89.25%).
The after panel is a composite, not a fresh 400-episode run: it retains 310 unchanged episodes, uses 70 first-revision episodes, and replaces two tasks with 20 final-revision episodes. Every revised task contributes all ten original initial states, including failures.
Small wording changes, more explicit goals
LIBERO’s fixed task sentences make the repair direct. We revised nine instructions using a small number of ordinary words: identify the blue-and-yellow can, ask for a drawer to open fully, or specify the back compartment between the two large side compartments.
The refining agent was constrained to clarify the goal rather than pass a detailed manipulation plan to the acting agent. The final text does not give exact coordinates, centimeter-level target positions, or control sequences. The table preserves the instructions verbatim and highlights added or changed words.
| Task | Original instruction | Improved instruction | Native successes |
|---|
Instruction comparisons in the recordings
These seven pairs use matching initial states and seeds. Each shows the lowest rollout index exhibiting its stated transition; the heading reports the full ten-episode task result. The selection illustrates the diagnosis rather than providing a random sample.
Full recordings at 20 fps, with scene and wrist views. Thinking time is omitted. Native success may stop the recording before release.
The spatial bowl task keeps the interpretation honest. Adding in the center of the plate reduces rejected completion claims from three to zero, but native success falls from 5/10 to 4/10. A clearer goal can improve agreement about completion while remaining difficult to execute.
3.2 RoboTwin: repair the instruction-generation chain
Across the available terminal results, IAS rises from 73.95% to 90.56%: completion claims rejected by the checker fall from 130/499 to 47/498. Native success rises from 320/499 to 396/498 (64.13% → 79.52%). On the 497 shared terminal episode IDs, 96 previous disagreements disappear, 32 remain, and 15 new ones appear. A disagreement disappearing does not necessarily mean the task becomes successful.
The instruction must follow the generated scene
RoboTwin’s task text can vary with the sampled object, side, or scene configuration. Editing one example sentence would leave the next generated episode untouched. The intervention therefore belongs in the generation chain: combine the episode-specific instruction with a task-level goal specification that spells out the required final relations.
The recorded rerun adds a public criteria block and a completion checklist. Depending on the task, this specifies object identity, orientation, position, containment, release, or conditions that must hold simultaneously. Some criteria include numeric tolerances or quaternion targets. This is a broader instruction supplement than LIBERO’s few-word edits; the minimality is in the layer changed, not a claim that every generated prompt is short.
The three cases below show the archived instructions and successful revised recordings, with the same base instruction and effective seed in each pair. Some other episodes use different effective seeds across runs, so RoboTwin’s aggregate before/after comparison is not a globally matched-scene estimate of wording alone.
The after table contains 498 terminal results from 500 planned episodes, including 19 infrastructure-error results. The gallery contains 479 playable scored videos: 396 successes and 83 failures, or 82.67% native success within that subset. Two episodes lack terminal results. The table and gallery answer different accounting questions; their denominators should not be interchanged.
RoboTwin results and coverage · Verbatim instruction comparisons and provenance · Full RoboTwin gallery ↗
04 / WHAT REMAINS DIFFICULT
Contact-rich manipulation
still challenges zero-shot control.
GPT-6-Astra solves many of the tasks in our evaluation, but its success is uneven. Repeated rollouts across 40 LIBERO tasks and 50 RoboTwin tasks let us identify recurring areas of weakness. Several tasks remain difficult when they require physical contact together with precise control of position or orientation.
Pushing a plate, closing a drawer after placement, or arranging objects in a confined space requires a sequence of small corrections. The robot must maintain useful contact, respond to how an object moves, and leave it in a stable final pose. These demands recur in the low-success tasks below.
Maintain useful contact
During pushing and drawer motion, a small change in contact can change the object’s response. Losing contact or approaching from an ineffective angle requires repositioning before progress can continue.
Control the final pose
Placement depends on position and orientation together. An object must settle correctly as it meets the table or container and the gripper releases it; reaching the vicinity of the target is only one part of that motion.
Coordinate successive contacts
Putting a bowl away and then closing the drawer, or placing two shoes in one box, requires several interactions to work in sequence. Each correction must preserve the progress already made.
LIBERO: pushing and articulated contact
The revised plate-pushing task succeeds in 2/10 rollouts; placing a bowl in the bottom drawer and closing it succeeds in 0/10. Both selected episodes reach the 500-step control budget. The plate remains short of its target, and the drawer remains open. These examples show how repeated contact adjustments can consume the available interaction budget.
Task rates use the final LIBERO composite. The plate uses the revised instruction; the drawer task retains its original instruction and baseline recordings.
RoboTwin: precise placement and release
The goal-spec rerun succeeds in 0/10 rollouts on can-and-pot placement and 4/10 on placing both shoes in a box. The can task combines rotation, relative placement, and release onto the table. The shoe task requires two objects to fit the box with the intended positions and orientations. Together, they illustrate the difficulty of controlling the final pose through contact and release.
Each rate covers all ten rollouts of its task. The videos show selected failed episodes; the task requirements and repeated outcomes motivate the pattern, without isolating a single physical cause for every failure.
Contact-rich precision is a recurring weak point in this agent’s zero-shot robot control. The low task-level success rates show that the challenge extends beyond an isolated bad rollout. Reliable performance requires continual visual feedback, careful contact adjustments, and recovery from small errors throughout the motion. For this agent and these harnesses, that capability remains inconsistent even when the intended goal is stated clearly.
05 / CONCLUSION
Repair the language.
Preserve the demonstrations.
Small instruction edits can yield substantial gains in benchmark alignment. In LIBERO, the repair is a few words in fixed task text. In RoboTwin, it is a goal-specification supplement to generated instructions. Both preserve the native checker and existing demonstration trajectories.
The practical value is a less disruptive way to improve an established benchmark: make its intended goal more legible before changing the task or collecting the data again. Existing recordings can be reused, with their language annotations updated consistently where needed.
GPT’s current language, vision, and tool-use abilities make it useful as a human-proxy policy for this work. It is not a ground-truth policy, and its claims require scrutiny. It can nevertheless generate informative disagreements, help formulate instruction repairs, and test whether those repairs improve agreement with the evaluator. That is already a useful role in debugging benchmarks.
Evidence, interpretation limits, and reproducibility
The authors developed the revisions using source inspection, demonstrations, and observed failures. This is an adaptive diagnostic intervention, not a held-out causal estimate. LIBERO’s after panel reuses unchanged baseline episodes; RoboTwin has missing results, infrastructure errors, and some changed effective seeds. IAS depends on completion-reporting behavior, and the instruction supplements differ in information content. We did not conduct a controlled human-operator baseline or a post-training/generalization experiment.
LIBERO reviewed judgments · LIBERO task instructions and run fingerprints · All 400 paired LIBERO records · LIBERO clip selection and hashes · RoboTwin outcomes and coverage · RoboTwin instruction versions and source references · Detailed LIBERO accounting.
The examples are selected to explain mechanisms. Videos retain recorded frames; thinking time is omitted. Public data preserve outcomes and provenance without private model reasoning. The full evaluation gallery remains available on Hugging Face.
The comparison data could not load. Please refresh, or use the linked JSON and CSV records.