01 / INTRODUCTION
Motivation
Imagine asking a person to operate a robot using only a task instruction and live camera views. They can see the objects and control the robot, but they do not know the exact target positions or tolerances written into the benchmark’s success checker. Does the instruction give them enough information to understand what counts as success?
If the person fails to manipulate the robot, that alone tells us little about the instruction. The more revealing case is when their behavior reasonably fulfills the stated request, a human reviewer would consider the task complete, and the benchmark still rejects it. The code may be testing a precise, reproducible condition that the instruction never communicated: a particular compartment, a narrower placement region, or an unstated orientation.
That gap matters for more than evaluation. A demonstration may satisfy the hidden condition while its instruction describes a broader or different goal. The recording teaches a convention that the language does not explain.
Why mismatched supervision matters
Consider an already capable generalist policy with a useful understanding of everyday instructions. Post-training it on these pairs could push it toward benchmark-specific interpretations that conflict with that prior understanding. The model is rewarded for reproducing an unspoken convention, potentially at the expense of behavior that transfers to other settings.
This is our motivation, not a post-training result. We did not update policy weights or measure a loss of generalization. We instead ask a prior question: can we make the instruction and evaluator agree more closely while keeping the expensive recorded demonstrations usable?
Our approach is to use GPT as a proxy for an instruction-following human operator. Its ability to interpret language, inspect images, and interact through a robot harness makes this audit practical now. It is fallible, but its disagreements with a benchmark provide concrete cases to inspect and repair.
02 / METHOD AND BENCHMARKS
Run an operator.
Audit the disagreement.
We built control harnesses for two representative benchmarks: LIBERO, whose task instructions are fixed strings, and RoboTwin, whose instructions vary with the generated scene. GPT-6-Astra operates the robot from task language, live RGB observations, and robot proprioception.
All stages requested gpt-6-astra with high reasoning.
The baseline harnesses follow the task-information boundary commonly used in VLA/WAM evaluation: the policy does not receive expert demonstrations, success-checker code, hidden object poses, rewards, or native success labels. The adapters provide camera calibration and generic control guidance so the agent can use the robot interface; they do not give it a long task-specific solution. This is a common information boundary, rather than a claim that GPT and a VLA/WAM use identical observation or action interfaces.
| Benchmark | Coverage | Instruction layer | Intervention |
|---|---|---|---|
| LIBERO | 40 tasks · 10 fixed initial states each | One instruction shared across all episodes of a task | Minimally edit nine task instructions to add only missing key information. Rerun every episode of each revised task. |
| RoboTwin | 50 tasks · 10 planned rollouts each | A distinct, scene-dependent instruction for each episode | Minimally edit instructions for episodes with agent–benchmark disagreements to add only missing key information. Rerun only those episodes. |
We first evaluated the original instructions and compared the agent’s completion judgment with the benchmark’s verdict. Some rejected rollouts appeared to satisfy a reasonable reading of the instruction. We then inspected the videos, task definitions, and demonstration references to identify missing goal information. The audit can use those sources; the acting policy does not receive them directly. The revised instruction is the channel through which the repair reaches the policy.
Read the instruction.
Observe the scene.
GPT controls the robot without demonstration videos or direct access to the success checker.
Looks complete.
Scores as failure.
Review disagreements to separate missing goal information from mistakes by the agent.
Clarify what
completion requires.
Make minimal instruction edits, then rerun the affected tasks or episodes.
Keep the checker.
Reuse the recordings.
Improve language–goal alignment without changing the native task or recollecting demonstrations.
We manually reviewed every episode where the agent and benchmark disagreed, retaining only cases where the agent’s actions reasonably fulfilled the instruction. In these cases, a human evaluator would consider the task successful, even though the benchmark marked it as a failure.
Instinct-alignment score
We define the instinct-alignment score (IAS) to summarize how often a benchmark rejects an outcome the operator considers complete:
Higher IAS means fewer such disagreements under the stated accounting rule. It is not a replacement for native task success: an agent that never claims completion on a failed rollout can obtain 100% IAS without solving the task. We report the two-by-two judgment tables, native success, and videos together.
Repair the instruction layer
Changing the verifier can redefine which recorded trajectories count as valid demonstrations and force a broader data audit or recollection. We take the less disruptive route: keep the native task and checker fixed, and ask the instruction-refining agent to communicate their goal more clearly. The demonstration recordings remain unchanged; a training pipeline can reuse them with the corrected language rather than collect new trajectories.
03 / RESULTS
Fewer disagreements.
Better-aligned benchmarks.
After instruction repair, the agent’s completion judgments agree more often with the benchmark’s verdict in both LIBERO and RoboTwin. Native task success also improves overall, although individual tasks can still regress.
3.1 LIBERO
Across the 400-episode comparison, human-reviewed agent–benchmark disagreements fall from 47 to zero, raising IAS from 88.25% to 100%. Native success rises from 322/400 to 357/400 (80.50% → 89.25%): 35 more episodes pass the same benchmark checker.
Two task-level examples
Goal · Task 09 improves from 7/10 to 10/10 native successes, while Long · Task 05 improves from 0/10 to 10/10. The table shows these two examples; the overall results above include all 40 tasks.
| Task | Original instruction | Improved instruction | Native successes |
|---|
Before-and-after rollouts
LIBERO Goal · Task 09 · Episode 04 and LIBERO Long · Task 05 · Episode 01 both change from benchmark failure to success. Each pair starts from the same initial state and seed. Each card also reports the task’s result across all ten episodes.
Full recordings at 20 fps, with scene and wrist views. Thinking time is omitted. Native success may stop the recording before release.
Across these two tasks, agent–benchmark disagreements fall from 13 to zero, and both tasks reach 10/10 native successes. The overall comparison above includes every task and initial state, including tasks with smaller gains or regressions.
3.2 RoboTwin
All 50 tasks: 500 original episodes versus a composite retaining 369 original episodes and replacing the 131 episodes with agent–benchmark disagreements.
Two episode examples
Task 00 · Episode 02 and Task 32 · Episode 02 both change from benchmark failure to success. Each pair uses the same episode slot and effective seed. The table shows the recorded instructions, followed by the original and revised videos.
| Episode | Original instruction | Revised instruction | Bench outcome |
|---|
RoboTwin results, episode records and provenance · September 24 gallery snapshot · Full RoboTwin gallery ↗
04 / WHAT REMAINS DIFFICULT
Contact-rich manipulation
still challenges zero-shot control.
GPT-6-Astra solves many of the tasks in our evaluation, but its success is uneven. Repeated rollouts across 40 LIBERO tasks and 50 RoboTwin tasks let us identify recurring areas of weakness. Several tasks remain difficult when they require physical contact together with precise control of position or orientation.
Pushing a plate, closing a drawer after placement, or arranging objects in a confined space requires a sequence of small corrections. The robot must maintain useful contact, respond to how an object moves, and leave it in a stable final pose. These demands recur in the low-success tasks below.
Maintain useful contact
During pushing and drawer motion, a small change in contact can change the object’s response. Losing contact or approaching from an ineffective angle requires repositioning before progress can continue.
Control the final pose
Placement depends on position and orientation together. An object must settle correctly as it meets the table or container and the gripper releases it; reaching the vicinity of the target is only one part of that motion.
Coordinate successive contacts
Putting a bowl away and closing the drawer, tipping a container, or holding two objects for scanning requires several interactions to work in sequence. Each correction must preserve the progress already made.
LIBERO: pushing and articulated contact
Plate-pushing success rises from 0/10 to 2/10 (0% → 20%). Placing a bowl in the bottom drawer and closing it remains at 0/10 → 0/10 (0% → 0%); this task retains its original instruction. Both selected episodes reach the 500-step control budget. The plate remains short of its target, and the drawer remains open. These examples show how repeated contact adjustments can consume the available interaction budget.
Each rate covers all ten initial states, comparing the original run with the final LIBERO composite. The plate uses the revised instruction; the drawer task retains its original instruction and baseline recordings.
RoboTwin: pouring and coordinated scanning
Emptying the bin remains at 1/10 → 1/10 native successes (10% → 10%), while scanning the tea box changes from 1/10 to 2/10 (10% → 20%). Success remains low in both tasks after instruction repair. Pouring requires controlling a container as its contents move; scanning requires holding and aiming two objects together.
Task 06 · Episode 02 and Task 41 · Episode 03 show the current gallery’s revised recordings, both marked as failures. Each rate covers all ten episode slots before and after repair; the after comparison uses reruns where available and retains unchanged originals. The scores alone do not identify the physical cause of a failure.
Contact-rich precision is a recurring weak point in this agent’s zero-shot robot control. The low task-level success rates show that the challenge extends beyond an isolated bad rollout. Reliable performance requires continual visual feedback, careful contact adjustments, and recovery from small errors throughout the motion. For this agent and these harnesses, native success remains low on these tasks after instruction repair.
05 / CONCLUSION
Repair the language.
Preserve the demonstrations.
Small instruction edits can yield substantial gains in benchmark alignment. In both LIBERO and RoboTwin, the edits add only missing key information while preserving the native checker and existing demonstration trajectories.
We also evaluated GPT-6-Astra on RoboCasa365 and RoboDojo, running one episode per task in each benchmark.
The practical value is a less disruptive way to improve an established benchmark: make its intended goal more legible before changing the task or collecting the data again. Existing recordings can be reused, with their language annotations updated consistently where needed.
GPT’s current language, vision, and tool-use abilities make it useful as a human-proxy policy for this work. It is not a ground-truth policy, and its claims require scrutiny. It can nevertheless generate informative disagreements, help formulate instruction repairs, and test whether those repairs improve agreement with the evaluator. That is already a useful role in debugging benchmarks.
The comparison data could not load. Please refresh, or use the linked JSON and CSV records.