01 / INTRODUCTION
Motivation
Imagine asking a person to operate a robot using only a task instruction and live camera views. They can see the objects and control the robot, but they do not know the exact target positions or tolerances written into the benchmark’s success checker. Does the instruction give them enough information to understand what counts as success?
If the person fails to manipulate the robot, that alone tells us little about the instruction. The more revealing case is when their behavior reasonably fulfills the stated request, a human reviewer would consider the task complete, and the benchmark still rejects it. The code may be testing a precise, reproducible condition that the instruction never communicated: a particular compartment, a narrower placement region, or an unstated orientation.
That gap matters for more than evaluation. A demonstration may satisfy the hidden condition while its instruction describes a broader or different goal. The recording teaches a convention that the language does not explain.
Why mismatched supervision matters
Consider an already capable generalist policy with a useful understanding of everyday instructions. Post-training it on these pairs could push it toward benchmark-specific interpretations that conflict with that prior understanding. The model is rewarded for reproducing an unspoken convention, potentially at the expense of behavior that transfers to other settings.
This is our motivation, not a post-training result. We did not update policy weights or measure a loss of generalization. We instead ask a prior question: can we make the instruction and evaluator agree more closely while keeping the expensive recorded demonstrations usable?
Our approach is to use GPT as a proxy for an instruction-following human operator. Its ability to interpret language, inspect images, and interact through a robot harness makes this audit practical now. It is fallible, but its disagreements with a benchmark provide concrete cases to inspect and repair.
02 / METHOD AND BENCHMARKS
Run an operator.
Audit the disagreement.
We built control harnesses for two representative benchmarks: LIBERO, whose task instructions are fixed strings, and RoboTwin, whose instructions vary with the generated scene. GPT-6-Astra operates the robot from task language, live RGB observations, and robot proprioception.
All stages requested gpt-6-astra with high reasoning.
The baseline harnesses follow the task-information boundary commonly used in VLA/WAM evaluation: the policy does not receive expert demonstrations, success-checker code, hidden object poses, rewards, or native success labels. The adapters provide camera calibration and generic control guidance so the agent can use the robot interface; they do not give it a long task-specific solution. This is a common information boundary, rather than a claim that GPT and a VLA/WAM use identical observation or action interfaces.
| Benchmark | Coverage | Instruction layer | Intervention |
|---|---|---|---|
| LIBERO | 40 tasks · 10 fixed initial states each | One instruction shared across all episodes of a task | Minimally edit nine task instructions to add only missing key information. Rerun every episode of each revised task. |
| RoboTwin | 50 tasks · 10 planned rollouts each | A distinct, scene-dependent instruction for each episode | Minimally edit instructions for episodes with agent–benchmark disagreements to add only missing key information. Rerun only those episodes. |
We first evaluated the original instructions and compared the agent’s completion judgment with the benchmark’s verdict. Some rejected rollouts appeared to satisfy a reasonable reading of the instruction. We then inspected the videos, task definitions, and demonstration references to identify missing goal information. The audit can use those sources; the acting policy does not receive them directly. The revised instruction is the channel through which the repair reaches the policy.
Read the instruction.
Observe the scene.
GPT controls the robot without demonstration videos or direct access to the success checker.
Looks complete.
Scores as failure.
Review disagreements to separate missing goal information from mistakes by the agent.
Clarify what
completion requires.
Make minimal instruction edits, then rerun the affected tasks or episodes.
Keep the checker.
Reuse the recordings.
Improve language–goal alignment without changing the native task or recollecting demonstrations.
We manually reviewed every episode where the agent and benchmark disagreed, retaining only cases where the agent’s actions reasonably fulfilled the instruction. In these cases, a human evaluator would consider the task successful, even though the benchmark marked it as a failure.
Instinct-alignment score
We define the instinct-alignment score (IAS) to summarize how often a benchmark rejects an outcome the operator considers complete:
Higher IAS means fewer such disagreements under the stated accounting rule. It is not a replacement for native task success: an agent that never claims completion on a failed rollout can obtain 100% IAS without solving the task. We report the two-by-two judgment tables, native success, and videos together.
Repair the instruction layer
Changing the verifier can redefine which recorded trajectories count as valid demonstrations and force a broader data audit or recollection. We take the less disruptive route: keep the native task and checker fixed, and ask the instruction-refining agent to communicate their goal more clearly. The demonstration recordings remain unchanged; a training pipeline can reuse them with the corrected language rather than collect new trajectories.
03 / RESULTS
Fewer disagreements.
Better-aligned benchmarks.
After instruction repair, the agent’s completion judgments agree more often with the benchmark’s verdict in both LIBERO and RoboTwin. Native task success also improves overall, although individual tasks can still regress.
3.1 LIBERO
Before · original instructions
n = 40088.25%
Instinct-alignment score
1 − 47 / 400 = 88.25%
| Agent’s completion judgment | Bench judges success | Bench judges failure |
|---|---|---|
| Agent considers complete | 322 | 47 |
| Agent does not consider complete | \ | 31 |
Bench judges success: 322/400 · 80.50%
After · revised composite
n = 400100.00%
Instinct-alignment score
1 − 0 / 400 = 100.00%
| Agent’s completion judgment | Bench judges success | Bench judges failure |
|---|---|---|
| Agent considers complete | 357 | 0 |
| Agent does not consider complete | \ | 43 |
Bench judges success: 357/400 · 89.25%
All 40 tasks: 400 original episodes versus a composite preserving 310 original episodes and replacing all 90 episodes from the nine revised tasks.
Across the 400-episode comparison, human-reviewed agent–benchmark disagreements fall from 47 to zero, raising IAS from 88.25% to 100%. Native success rises from 322/400 to 357/400 (80.50% → 89.25%): 35 more episodes pass the same benchmark checker.
Two task-level examples
Goal · Task 09 improves from 7/10 to 10/10 native successes, while Long · Task 05 improves from 0/10 to 10/10. The table shows these two examples; the overall results above include all 40 tasks.
| Task | Original instruction | Improved instruction | Native successes |
|---|---|---|---|
| Goal t09 | put the wine bottle on the rack | put the wine bottle on the upper rack, with its base against the lower rail | 7/10 → 10/10+3 successes |
| Long t05 | pick up the book and place it in the back compartment of the caddy | pick up the book and place it in the back compartment of the caddy, between the two large side compartments | 0/10 → 10/10+10 successes |
Before-and-after rollouts
LIBERO Goal · Task 09 · Episode 04 and LIBERO Long · Task 05 · Episode 01 both change from benchmark failure to success. Each pair starts from the same initial state and seed. Each card also reports the task’s result across all ten episodes.
Full recordings at 20 fps, with scene and wrist views. Thinking time is omitted. Native success may stop the recording before release.
LIBERO Goal · Task 09 · EPISODE 04
A failure becomes a success
“put the wine bottle on the rack”
“put the wine bottle on the upper rack, with its base against the lower rail”
At initial state 3, the original rollout is judged complete by the agent but rejected by the benchmark; the revised rollout passes. Across all ten initial states, native successes increase from 7/10 to 10/10.
LIBERO Long · Task 05 · EPISODE 01
A failure becomes a success
“pick up the book and place it in the back compartment of the caddy”
“pick up the book and place it in the back compartment of the caddy, between the two large side compartments”
At initial state 0, the original rollout is judged complete by the agent but rejected by the benchmark; the revised rollout passes. Across all ten initial states, native successes increase from 0/10 to 10/10.
Across these two tasks, agent–benchmark disagreements fall from 13 to zero, and both tasks reach 10/10 native successes. The overall comparison above includes every task and initial state, including tasks with smaller gains or regressions.
3.2 RoboTwin
Before · original RoboTwin instructions
n = 50073.80%
Instinct-alignment score
1 − 131 / 500 = 73.80%
| Agent’s completion judgment | Bench judges success | Bench judges failure |
|---|---|---|
| Agent considers complete | 336 | 131 |
| Agent does not consider complete | \ | 33 |
Bench judges success: 336/500 · 67.20%
After · revised composite
n = 50093.00%
Instinct-alignment score
1 − 35 / 500 = 93.00%
| Agent’s completion judgment | Bench judges success | Bench judges failure |
|---|---|---|
| Agent considers complete | 430 | 35 |
| Agent does not consider complete | \ | 35 |
Bench judges success: 430/500 · 86.00%
All 50 tasks: 500 original episodes versus a composite retaining 369 original episodes and replacing the 131 episodes with agent–benchmark disagreements.
In the latest gallery results, native success rises from 336/500 to 430/500 (67.20% → 86.00%). Of the 131 revised episodes, 94 now pass the benchmark; 37 remain unsuccessful, including 1 timeout.
Agent–benchmark disagreements fall from 131 to 35, raising IAS from 73.80% to 93.00%.
Two episode examples
Task 00 · Episode 02 and Task 32 · Episode 02 both change from benchmark failure to success. Each pair uses the same episode slot and effective seed. The table shows the recorded instructions, followed by the original and revised videos.
| Episode | Original instruction | Revised instruction | Bench outcome |
|---|---|---|---|
| Task 00EPISODE 02 | Position the bottle with white neck and red top head-up and lift it | Lift the bottle with the white neck and red top head-up, keep it upright, and move it to the right side of the table. | failure → success |
| Task 32EPISODE 02 | Pick up the toycar with transparent windshield and set it inside the bright yellow basket. | Pick up the toy car with the transparent windshield, place it inside the bright yellow basket, and then lift the basket with the toy car inside. | failure → success |
RoboTwin · Task 00 · EPISODE 02
Adjust bottle
“Position the bottle with white neck and red top head-up and lift it”
“Lift the bottle with the white neck and red top head-up, keep it upright, and move it to the right side of the table.”
RoboTwin · Task 32 · EPISODE 02
Place object basket
“Pick up the toycar with transparent windshield and set it inside the bright yellow basket.”
“Pick up the toy car with the transparent windshield, place it inside the bright yellow basket, and then lift the basket with the toy car inside.”
RoboTwin results, episode records and provenance · September 24 gallery snapshot · Full RoboTwin gallery ↗
04 / WHAT REMAINS DIFFICULT
Contact-rich manipulation
still challenges zero-shot control.
GPT-6-Astra solves many of the tasks in our evaluation, but its success is uneven. Repeated rollouts across 40 LIBERO tasks and 50 RoboTwin tasks let us identify recurring areas of weakness. Several tasks remain difficult when they require physical contact together with precise control of position or orientation.
Pushing a plate, closing a drawer after placement, or arranging objects in a confined space requires a sequence of small corrections. The robot must maintain useful contact, respond to how an object moves, and leave it in a stable final pose. These demands recur in the low-success tasks below.
Maintain useful contact
During pushing and drawer motion, a small change in contact can change the object’s response. Losing contact or approaching from an ineffective angle requires repositioning before progress can continue.
Control the final pose
Placement depends on position and orientation together. An object must settle correctly as it meets the table or container and the gripper releases it; reaching the vicinity of the target is only one part of that motion.
Coordinate successive contacts
Putting a bowl away and closing the drawer, tipping a container, or holding two objects for scanning requires several interactions to work in sequence. Each correction must preserve the progress already made.
LIBERO: pushing and articulated contact
Plate-pushing success rises from 0/10 to 2/10 (0% → 20%). Placing a bowl in the bottom drawer and closing it remains at 0/10 → 0/10 (0% → 0%); this task retains its original instruction. Both selected episodes reach the 500-step control budget. The plate remains short of its target, and the drawer remains open. These examples show how repeated contact adjustments can consume the available interaction budget.
Each rate covers all ten initial states, comparing the original run with the final LIBERO composite. The plate uses the revised instruction; the drawer task retains its original instruction and baseline recordings.
Goal t05
Sustained contact while pushing a plate
The plate must slide into a small region near the stove. The policy repeatedly repositions the gripper but does not finish within 500 control steps: sustained contact and small positional corrections remain difficult.
“push the plate to the front of the stove, close to its front edge”
Long t03
Placement followed by drawer closure
The bowl reaches the drawer, but the drawer remains open after 500 control steps. The task requires a change of contact after placement, followed by a controlled push to complete the closure.
“put the black bowl in the bottom drawer of the cabinet and close it”
RoboTwin: pouring and coordinated scanning
Emptying the bin remains at 1/10 → 1/10 native successes (10% → 10%), while scanning the tea box changes from 1/10 to 2/10 (10% → 20%). Success remains low in both tasks after instruction repair. Pouring requires controlling a container as its contents move; scanning requires holding and aiming two objects together.
Task 06 · Episode 02 and Task 41 · Episode 03 show the current gallery’s revised recordings, both marked as failures. Each rate covers all ten episode slots before and after repair; the after comparison uses reruns where available and retains unchanged originals. The scores alone do not identify the physical cause of a failure.
RoboTwin · Task 06 · EPISODE 02
Emptying a bin onto the tabletop
The instruction asks the robot to lift and tip the bin so its balls pour onto the tabletop. This motion requires control of the bin while its contents move.
Task instruction“Lift the simple white plastic trash bin high and tip it over onto the table so the balls pour out on the tabletop.”
RoboTwin · Task 41 · EPISODE 03
Holding and aiming two objects
The two arms must hold the tea box and scanner at the same time, then aim the scanner face at the box. This requires coordinating the positions and orientations of both objects.
Task instruction“Grab the smooth black tea box and the scanner with the flat reading surface at the same time, then point the scanner face at the tea box as if scanning it.”
Contact-rich precision is a recurring weak point in this agent’s zero-shot robot control. The low task-level success rates show that the challenge extends beyond an isolated bad rollout. Reliable performance requires continual visual feedback, careful contact adjustments, and recovery from small errors throughout the motion. For this agent and these harnesses, native success remains low on these tasks after instruction repair.
05 / CONCLUSION
Repair the language.
Preserve the demonstrations.
Small instruction edits can yield substantial gains in benchmark alignment. In both LIBERO and RoboTwin, the edits add only missing key information while preserving the native checker and existing demonstration trajectories.
We also evaluated GPT-6-Astra on RoboCasa365 and RoboDojo, running one episode per task in each benchmark.
The practical value is a less disruptive way to improve an established benchmark: make its intended goal more legible before changing the task or collecting the data again. Existing recordings can be reused, with their language annotations updated consistently where needed.
GPT’s current language, vision, and tool-use abilities make it useful as a human-proxy policy for this work. It is not a ground-truth policy, and its claims require scrutiny. It can nevertheless generate informative disagreements, help formulate instruction repairs, and test whether those repairs improve agreement with the evaluator. That is already a useful role in debugging benchmarks.
These results suggest a path toward self-improving benchmarks. Agents could repeatedly diagnose instruction–verifier gaps, propose minimal repairs for human review, and retest the revised tasks while preserving the native verifier and demonstration trajectories. Better agents could help improve evaluations, while clearer evaluations could provide more reliable feedback for future agent development. This feedback loop could potentially support recursive self-improvement (RSI). Evaluating it across successive generations of agents is a direction for future work.
Citation
Please cite this work as:
Zhang, Tianqi, Yitong Li, and Mingyu Ding. “Agent-Guided Instruction Repair for Robotics Benchmarks”. (Sep 2026). https://tianqi-zh.github.io/BenchMend/
Or use the BibTeX citation:
@article{zhang2026benchmend,
title = {Agent-Guided Instruction Repair for Robotics Benchmarks},
author = {Zhang, Tianqi and Li, Yitong and Ding, Mingyu},
journal = {BenchMend},
year = {2026},
month = {September},
url = {https://tianqi-zh.github.io/BenchMend/}
}