Just a VLM Agent Can Play Robots
Show Lab, National University of Singapore
*Equal contribution · †Corresponding author
One interface across scenes, tasks and embodiments. Frontier VLMs hold the top band at seconds per step, and the same interface fine-tuned onto open-source backbones holds it from 12 Hz up to 33 Hz — while π0.5 and GR00T sit a band below at 20 and 24 Hz.
Foundation vision–language models (VLMs) exhibit broad intelligence about the world, yet translating this intelligence into robot control remains challenging. We present Show-Harness, an Embodied Harness that enables VLMs to “play” robots through a compact semantic interface linking intent to action. Show-Harness exposes discrete semantic action units that VLMs can naturally reason over, while embodiment-specific interpreters deterministically ground them into local robot actions, keeping the VLM directly responsible for fine-grained physical decisions. Through the same interface, Show-Harness demonstrates the feasibility of (1) directly unlocking closed-source frontier VLMs for zero-shot robot control, and (2) adapting small-scale open-source VLMs for low-cost deployment with just a few GPU-hours of fine-tuning. We further develop GUMI (GUI Manipulation Interface), which extends the same semantic action space to GUI-based demonstration collection, allowing humans and agents to “play” robots across embodiments without specialized teleoperation hardware. Extensive experiments show that Show-Harness-equipped VLM agents generalize robustly across tasks, embodiments, and environments, outperforming representative agentic and VLA paradigms. These results suggest that the right interface can unlock substantial embodied capability from foundation VLMs, without requiring additional model capacity or costly embodiment-specific pretraining.
Show-Harness turns a VLM into a situated agent: it perceives, reasons, and acts through a compact set of semantic action units that embodiment-specific interpreters ground into robot motion.
Robot control is reformulated into parameter-free units that a VLM can reason about directly. Each unit is a bare symbol; the interpreter supplies its metric magnitude, so the same vocabulary drives a 7-DoF Franka, a 6-DoF AgileX, and a simulator without changing the model.
A unit is a bare symbol — the interpreter supplies its metric magnitude. Pick one to see what it does.
A unit like MV_LEFT carries a name the model already knows plus a convention
fixing whose left it is. Arbitrary symbols A–F work if the convention
is spelled out (95% vs 100%); the names alone, unexplained, still reach 90%.
Remove both and the agent has to infer the mapping from its own interaction — success collapses to 5%, with 19 of 20 episodes timing out.
Asked to write that mapping down, the model gets 23.3% of six directions right against a 16.7% chance rate, and 11 of 20 trials mirror left and right. The interface is not decoration around the action space; it is the action space.
Each step closes the loop: plugins assemble the context, the VLM emits one semantic action unit, and the interpreter grounds it into motion.
Every action unit is a labeled key, so demonstrating a task amounts to playing the robot. A human at the keyboard and a GUI agent reading the same screen drive the robot identically, and every recorded step is a ready training pair — no specialized teleoperation hardware required.
A GUI agent driving the dual-arm robot through GUMI, one action unit per keystroke.
Two routes, one interface: a zero-shot agent (ZS) on Gemini-3.1 Pro, and a fine-tuned Qwen3.5-2B (FT). Both beat all three baseline families — VLA (π0.5, GR00T), VLA-centric agents (H-VLA, G-VLA) and code-as-policy (CaP-X, RATS).
ZS clears every environment shift at 20/20, including the Teddy and Chess tasks held out from fine-tuning. Trained only in simulation, FT still transfers to the real robot at 13/20 — both trainable VLA baselines score 0/20.
Adapting to a new physical demand is a change to the interpreter, not to the model — a smaller step, an extra unit, a joint prediction over two arms.
Physical. A 1 cm interpreter step lifts ZS 60% → 80% and FT
40% → 65% with no retraining, where π0.5 needs its entire budget on fine-grained data
to reach 60%. Rotation units extrapolate to an unseen 90° (ZS 10/10, π0.5 20%), and joint
dual-arm prediction lifts the average 50% → 80% while removing every collision.
Semantic. Situated Planning takes ZS to 85% where FT alone gets 10%. A single video
demonstration has ZS follow the demonstrated order 20/20, against 20% with nothing to watch.
Among frontier VLMs with strong reasoning, zero-shot robot control is limited by fine-grained spatial grounding, not by language understanding or planning.
Five frontier VLMs under one harness split into two tiers: Gemini-3.1 Pro, GPT-5.6-sol and Opus 5 at 86–96%, GPT-5.6-luna and Gemini-3.6-flash at 72–78%. Thinking effort mainly shortens episodes — 37 to 30 steps for GPT-5.6-sol, at 3.4× the wall-clock. Planning is not the gap: instruction following (≥98%) and subtask plans (≥97%) are near-saturated everywhere, while empty grasps climb from 6% to 26% down the ranking. The models miss on the centimetres.
Above 2B, capacity buys precision and costs latency — and on a moving target the latency wins.
Success is flat from 2B up — Qwen3.5 at 2B/4B/9B and InternVL3.5 at 2B/8B all within four points (88–92%). The two smallest backbones emit valid units but hesitate, stretching episodes from 37.6 to 57.8 steps at 72–74%. Capacity pays where precision is the bottleneck: stacking and peg insertion go 50% → 65% → 75% from 0.8B to 9B. The one reversal is the rolling tennis ball, where 2B (80%) beats 0.8B (70%) and 9B (60%) — at 79 ms per decision the 9B model cannot track a moving target, against 39 ms for 2B.
Default plugins buy raw success; optional ones buy reach on the cases that need them, at no cost to the rest.
Removing one default plugin at a time from the full 96% configuration: Multi-View Guidance is worth 38 points, Subtask Planning 36, Proprioception 28, Failure Recovery 24, and Action History 20 while cutting 23% of the steps. The rest trade cost for reach: Action Chunking holds 96% with 19% fewer VLM calls, and Adaptive Step reaches 96% in 30 steps, beating a fine 2 cm step (82%) and a coarse 4 cm one (70%). Visual Prompt and Situated Planning stay off by default, each lifting its target case from 35% to 85%.
@misc{chen2026showharnessjustvlmagent,
title = {Show-Harness: Just a VLM Agent Can Play Robots},
author = {Yanzhe Chen and Zechen Bai and Zhijun Cao and Wenzheng Zeng and Kevin Qinghong Lin and Yiqi Lin and Guoqiang Liang and Kevin Yuchen Ma and Qiming Huang and Mike Zheng Shou},
year = {2026},
eprint = {2609.10522},
archivePrefix = {arXiv},
primaryClass = {cs.RO},
url = {https://arxiv.org/abs/2609.10522}
}