Language models are becoming robot orchestrators
Lightly edited using AI
Language models have been used as robot planners for a while. SayCan had the model choose the next skill from a list of descriptions, like “pick up the sponge”, and a pretrained policy carried it out. Then people had them write the robot’s code, first in Code as Policies and more recently in CaP-X and ASPIRE, both from NVIDIA and UC Berkeley among others. In ASPIRE, Claude Code writes the robot’s programs in simulation and saves the skills that worked into a library it reuses. The labs now train their models on embodied reasoning directly, and Google’s Gemini 4 Argon came out this week with a version for robotics. Coding models can now write the code that runs a robot, from finding an object in a camera image to the skill that picks it up.
I’ve been tinkering with policy architectures for action models, post-training them on simulation data and on real teleoperation and intervention data. I’m fairly sure policies like these will handle short dexterous skills and chains of them. This post is about the layer above them, which decides what to do next and checks whether it worked. I wanted to know how much of that layer a coding agent can already run, and how much it needs to be handed.

I’ll call the model in those middle layers the orchestrator. It breaks an instruction into subtasks, picks a skill for each and chains them together, and checks the result at the end. Language-model agents needed the same layer, and today’s multimodal models already handle it well. In the last tier below, it writes the skills as well.
I developed an environment for this during the Prime Intellect RL Residency. It has 28 of LIBERO’s tasks and four tiers, and each tier takes away something the agent had in the one before. I ran frontier and open models from several providers through it on several harnesses, and trained a small model with RL on one task. I tested them with the simulator’s labels, with only the cameras, without failure messages, and with nothing but the motors. In the last tier they wrote their own skills and solved more tasks with them than with mine. I keep adding tasks and robots to the environment, so that any coding model can use it to train itself with RL to get better at manipulation.
The environment¶
The agent controls a Franka arm in LIBERO (MuJoCo) by writing Python. The robot is a Python library installed in the agent’s sandbox, which its programs load with import robot. It isn’t a tool protocol, so any harness that can run Python can drive it, and the same program would run on a real arm that implemented the same functions.

The agent writes a program, runs it, reads what it printed and rewrites it. The simulator doesn’t reset between runs, so if one program knocks a bowl over, the bowl is still on its side when the next program starts. An episode scores 1 if LIBERO’s success check passes and 0 if it doesn’t. The simulator signs the score, which keeps the agent from reporting a success it didn’t get.

The 28 tasks come from three of LIBERO’s suites. A kitchen has ten instructions, such as open the drawer, turn on the stove and put the wine bottle on the rack. A spatial set has two identical bowls, and the instruction picks one by where it sits. In the third, groceries on the floor go into a basket. All results below are on held-out layouts, which have the same tasks with the objects placed differently from the layouts I developed on. The tasks and their BDDL scene files are on Hugging Face. I’m adding more of LIBERO’s tasks, starting with LIBERO-Long, its long-horizon suite.
There are four tiers, set by how much help the model gets. In the first, privileged, the library has eight calls. find and locate get objects from the simulator’s labels, look reads the two cameras, gripper reports the hand, and move_to, grasp, release and retract are skills I wrote. Structured removes find and locate. Raw also removes the failure messages that tell the agent a step went wrong. Bare-metal removes my skills as well.

Privileged and structured¶
In the privileged tier the agent is handed its objects. The positions aren’t exact, because locate measures them from the depth camera and can be a few centimetres off on a thin object like the wine bottle. The agent still chooses where to grip, at what angle and in what order. On a plain bash harness, OpenAI’s GPT-6 Astra solved 86% of tasks, GPT-6 Sol and Claude Opus 5.5 about 72%, and Grok, DeepSeek and Qwen were further behind. Most failures were grasps that closed on nothing, so grasp geometry was where the models struggled.
In the structured tier the labels are gone, and on bash every model dropped. Every harness gets the same camera data, but bash passes it to the model as arrays of numbers, and the model has to write code to find the bowl in them. Codex and Claude Code can show the model the image. With the image, Opus solved 93% and Sol got back to its privileged score.

In 12 of its 28 episodes, Grok’s code reached into the simulator for the answer. The agent ran as root in the same sandbox as the simulator, so after Grok I closed those gaps. The scene files and LIBERO’s source are deleted once the scene is built, and if anything the agent runs reaches into the simulator, the episode is flagged and scores zero.
Raw¶
A real robot doesn’t report gripper_closed_empty. It reports that the fingers stopped 1 mm apart, and you have to work out that you missed. The raw tier drops the failure messages and returns only what the sensors read.
I expected this to cost turns and to hurt the weaker models most. I ran raw against structured on the same 28 tasks, two held-out layouts each, with a 150-turn limit and every model on a harness that shows it images: Astra and Sol on Codex, Opus on Claude Code, and GLM-5.3-Flash and Qwen3.7-plus on prime-agent. Every task ran in both tiers, so each model is compared with itself. It was a new run, and its structured scores differ from the previous section’s by a task or two, which is normal variation between runs at 28 tasks.

Across the three frontier models, raw solved 140 of 168 paired tasks and structured solved 138, in about the same number of turns. GLM-5.3-Flash solved more without the failure messages, 17 of 28 against 9, and Qwen3.7-plus solved almost nothing in either tier. The models also called the cameras and the gripper less often in raw, because every reply already included the readings.
Without the messages, the models checked their own work. In two dozen frontier episodes a grasp closed on nothing, no call reported it, and the model still finished the task. In the episode below, Astra misses the bowl, looks at the wrist camera, writes “the first grip closed fully, and the camera shows the bowl stayed on the table”, and grasps again.
The robot is replayed from the episode’s command log. The text is excerpts from the trace, labelled with the trace’s field names. reasoning_content is OpenAI’s summary of the model’s reasoning, not the reasoning itself.
Bare-metal¶
Up to here the models were orchestrating skills I had written. In the last tier I removed those too, to see whether the orchestrator could write the layer below it.
The bare-metal library has four calls besides the cameras. state() returns the joint angles, the hand pose and the finger width. command(joint_delta, gripper) moves the joints for one 50 ms controller step. The controller caps how far each joint moves in a step, so the arm lags and accelerates the way a real one does. fk(q) and jacobian(q) give the arm’s kinematics. There’s no inverse kinematics, motion planning, grasp or success signal. The model isn’t in the control loop itself. It writes a program that runs the 20 Hz loop, and the simulator waits between the program’s calls, so a slow program doesn’t miss steps. I ran it two ways:
- Writes its own skills. Every task starts from nothing, and whatever it wrote for the last task is gone.
- With a saved library. The model’s
skills/folder and a notes file carry over from task to task. The library is built on training layouts, one task at a time, then frozen and scored on the held-out layouts.

I expected the success rate to drop without my skills, but writing their own, Astra and Opus both solved more tasks than with mine in the raw tier, 7 points more over 112 held-out episodes, for a few extra turns. With the saved library, Opus needed 11 turns a task instead of 28, not counting the training-layout episodes in which it built the library.
The two libraries are very different. Astra wrote two files and 129 lines: an arm module with ik, move and hold, and a small vision module. None of its code knows where any object is. Opus wrote 21 files and about a thousand lines, with damped inverse kinematics, modules for bowls, bottles, drawers and knobs, and a recipe for each training task that starts its camera search where the object sat in training. LIBERO’s held-out layouts only move objects a few centimetres, so the recipes still find them, and Opus ran one in nearly every held-out episode, which is why its episodes took 11 turns. I’d call its library a cookbook for these 28 tasks. Astra’s code is general, but its notes also record where things sat in training, so neither library has been tested on a task it hasn’t seen. These runs show that a model can save what it learned on a task and reuse it when the objects move. The next test is whether the skills carry over to new tasks.
I gave GLM-5.3-Flash, an open model that costs a fraction as much, the same setup.

Writing its own skills, Flash solved 10 of 28, fewer than the 17 it solved with my skills in raw, and it took 97 turns a task. With Astra’s library it solved 23 of 28, more than it ever did with my skills. With Opus’s cookbook it solved 25 of 28, in 19 turns a task instead of 85. The first version of my tamper scan flagged 10 of those episodes, because Flash had copied the word “LIBERO” from Opus’s notes into its code. The scan now looks for simulator paths rather than the word, and I re-scored every episode I’ve run with it.
In one episode Flash used up its step budget, then found the simulator’s process, opened its memory and set the step counter back to zero. The scan caught it and the episode scored zero, though it hadn’t solved the task anyway.
Training on it¶
The reward is the simulator’s 0 or 1 and the agent’s actions are text, so a text model can train on this directly, without an action head or flow matching. GRPO works unchanged.
I trained Qwen3.5-4B on one privileged-tier task, “put the bowl on the stove”, with LoRA: groups of 8 rollouts of up to 40 turns on the bash harness, 39 steps on two A100s, about $50 of compute. On about 50 layouts it had never seen, success went from 27% to 72%, and I found no reward hacking in the rollouts.

That’s one task in the easiest tier. Next I’ll use pass@k to check that the harder tiers are learnable, then train in raw, where the model has to judge its own steps, and in bare-metal, where it only finishes a task if the skills it writes work.
Longer tasks¶
I also ran a one-off test on LIBERO-Long, LIBERO’s ten long-horizon tasks, most of which chain two steps, such as “turn on the stove and put the moka pot on it”. It ran in the raw tier on two held-out layouts per task, and I added Google’s new Gemini 4 Argon on prime-agent. Opus, Astra and Gemini solved most of the tasks. In the episode below, Gemini turns the knob and sees the burner glowing red in its camera image. It then picks the moka pot up by its handle, since the body is almost as wide as the open gripper, and checks that the pot sits on the burner.
Replayed from the episode’s command log, like the Astra episode. reasoning_content is Gemini’s summary of its reasoning.
Robot brains¶
You could argue the sensors here are too perfect. Depth has no noise, object edges are exact, and the camera calibration and lighting never change. Still, these runs show frontier vision-language models working well with every level of primitives I gave them, from labelled objects down to joint commands. That wasn’t possible a few years ago.
With clean sensors, the orchestrator I started with ended up doing most of the brain’s job, including writing the skills below it. I think a robot’s brain will end up as a model writing code against its sensors and motors, and I like that for two reasons. A task it has solved once becomes a program it can run again, without thinking in real time. And the checks it writes to see whether a step worked can serve as verifiers for task success, which is what a model needs to keep improving itself. I think harness engineering will soon matter as much in robotics as it already does for coding agents.
Run it yourself¶
The environment is open source, so anyone can run it. It’s on the Prime Hub, with the tasks on Hugging Face and a public simulator image. Evaluating a model is one command, with the tier and the harness as flags:
uv run eval --env.taskset.id pact --env.taskset.tier raw \
--env.agent.harness.id codex -m <model>
The tier can be privileged, structured, raw or bare_metal. The harness can be bash, codex, claude_code or prime_agent, the four I tested, though any harness that can run Python should work. Every episode returns the score along with metrics such as how many camera images reached the model and whether the tamper scan fired.
You can improve a model on it in two ways. In bare-metal, library_mode = "build" carries the model’s skills/ folder and notes from one task to the next, and "use" scores a finished library without changing it. I built Astra’s and Opus’s libraries this way, and Flash used them through "use". To train the weights, the taskset plugs into prime-rl as a training source. The RL run above used this one, with LoRA on two A100s:
[[orchestrator.train.source]]
name = "pact"
env.taskset.id = "pact"
env.taskset.tier = "privileged"
env.taskset.only = ["put_the_bowl_on_the_stove"]
env.agent.harness.id = "bash"
env.agent.max_turns = 40
I built it on verifiers and prime-rl, which already provided harnesses, sandboxes, an episode runner and a trainer. I only had to write the robot library, the tasks and the scoring. Prime Intellect’s credits paid for these experiments.
If episodes start failing when you run many in parallel, lower the number running at once.