can VLAs be steered towards success with better prompts?

this article has been cleaned using AI

i wrote about detecting when a robot policy is failing a while ago. once you have a signal about a rollout going wrong you want to either halt the robot, or find ways to steer it towards success.

the vanilla thing to do here is to use dagger, where you intervene wherever the policy fails (or is about to fail) and then use those human intervention steps to finetune the model towards a better action distribution. if you don’t want human intervention, you can probably bake in adaptive planning after every K steps, or use labelled rollouts to train a value function and then conditionally sample the action head output using it.

i was exploring whether this could happen in real time, without a teleop rig, without labelled rollouts, without a human anywhere in the loop. basically, if a robot is deployed in production and something starts going wrong, can it talk itself out of it?

so i trained a smaller model to learn what to say when it sees the policy fail. the corrector’s action is the instruction, the environment is the frozen policy running in sim, and the reward is whether the task succeeded, read straight off the benchmark’s own success predicate. no labels required and the corrector improves by watching a policy fail and finding out which sentences make it stop failing.

spent too much time fitting an out of distribution task and environment to a model¶

i placed an SO101 arm in MuJoCo, where it would pick a cube and place it in a tray. recorded 50 scripted demonstrations generated with damped least squares IK that ggand0/pick-101 demonstrated for the so101 arm in sim and then used molmoact2, finetuning its action head on the demos. the finetuning worked well (loss 0.65 → 0.02), and it went from hovering over the cube zero shot to actually grasping it.

the loop then looked like: frozen VLA runs, uncertainty spikes, a Qwen3-VL corrector sees the failure frames and emits a corrective instruction, the instruction goes back in, reward is sim success, train the corrector with GRPO on prime-rl.

technically, the machinery worked. reward flowed into advantages and into LoRA updates on a real corrector supervising a real policy in a real loop, and nearly every step had enough within-group variance to produce a gradient, so the weights moved on all 40 of them. they just moved nowhere :D

so i ran a quick diagnostic. freeze the policy at one observation, feed it two opposite instructions, and measure the gap between the joint angles it commands in each case, averaged over the 6 joints. flow matching is stochastic so i sample each instruction a few times and average before differencing, otherwise you’re measuring sampling noise and calling it language. i’d written the bands into the script before running it, under 2 degrees per joint means the policy barely responds and no amount of reward shaping will fix that, over 10 means it’s steerable and my RL setup is the problem.

it came out at 0.02 degrees per joint. the base model zero shot gave 0.01. for scale the demonstrations swing joints through tens of degrees, so the difference between “put the cube in the tray” and its opposite was too small for the arm to physically execute.

i later realised that i finetuned on a single instruction string, applied identically to all 3150 training frames. one task, one embodiment, one sentence. this essentially makes the gradient useless for language steering as the instruction is constant across the entire dataset, and a constant input carries zero information. the vision to action map trains fine, which is why the policy learned to grasp. but the instruction has zero variance across the dataset, and a feature that never varies can’t explain anything so language became redundant by construction.

i tried relabelling every demo with three sub-goal captions that change over the episode (“reach toward the cube and grasp it” -> “lift the cube and carry it to the tray” -> “place the cube in the tray and let go”), grew to 50->150 episodes, and ran two finetunes on identical data: one with the VLM frozen, one with LoRA on the VLM backbone. the second exists only to test if language needs to be passed further back the reasoning chain.

both landed at 0.00–0.01 degrees per joint, i.e. the gap between the joint angles it commands under instruction A vs instruction B at the same observation. the arm gets sent to the same pose either way. deaf to instruction.

i also ran a closed loop success check, since an action delta at a frozen state can’t see effects that accumulate over a trajectory. n=20 hinted at +15 to 20 points for the captions and it did not survive, at n=60 per condition it collapsed to +5 points, which at that sample size is well inside what you’d see from chance. the 150 episode models do beat the 50 episode one, but that’s 3x the data, not the captioning.

changing the caption over time doesn’t fix the underlying problem, because each caption’s time window still maps to one specific stage of the task that the camera can already see. reaching looks like reaching, carrying looks like carrying. the caption is predictable from the pixels, so the model can still ignore it and get the same loss.

tried testing language steering on LIBERO¶

two things were probably wrong with that setup. my sim looked nothing like what the model was trained on, and finetuning it that narrowly had cost it whatever generality it came with. so i went back to a general molmoact2 checkpoint and a benchmark somebody else built. LIBERO ships 130 manipulation tasks on a Franka arm in robosuite, with human demonstrations for each and, more importantly, a success criterion written as a logical predicate over simulator state rather than something i made up. the checkpoints are public and the instruction strings are the benchmark’s own. every steerability number i had until then came from one model, in a sim i built, on a task i defined, with instructions i wrote, judged by a rule i invented.

the probe was the same idea but led to a few more interesting findings. libero has scenes with two black bowls on a table, identical, the same object twice. the camera cannot tell you which one is meant. the only thing that can is the sentence.

so i froze the arm at the first frame and asked the policy what to do twice, once told to pick up the bowl on the stove and once told to pick up the bowl next to the cookie box. if it reads the instruction the two answers should send the arm to different places. if it doesn’t i get the same answer twice.

i got different answers, and different in the right direction. the way i scored it, call it alignment, was to check whether the change in the commanded motion pointed from the first bowl towards the second. 1 means it lines up perfectly, 0 means the two are unrelated, and i’d set 0.30 as the bar beforehand. it came out at 0.765. the policy is stochastic so asking the same question twice already gives slightly different answers, and swapping the words moved it about 96 times more than that.

there’s a simpler version of the same question. take a scene with six grocery items and a basket where the instruction names one of them, and run 60 episodes per version, changing only the noun.

taking the noun out drops it by 38 points. naming a different item that is genuinely sitting on the table, which is the third row, drops it by 95. the 60% is the more interesting number. with six items on the table a policy guessing blindly would get 17%, so when you stop telling it which object, it isn’t guessing. it’s falling back on something.

so the instruction is a real input to these policies. the problem earlier was the policy, not the channel.

what breaks these policies¶

if the robot fixes itself by writing a better instruction, the thing to measure is which failures a better instruction can fix. so the next question was what happens when the instruction is one the policy has never seen, which is the normal case in deployment, because nobody types the benchmark string. they type whatever they’d say to a person.

that turns out to depend entirely on what’s different about the sentence. saying the same thing a different way barely changes the success rate. i tried paraphrases, synonym swaps and shortened references, 240 episodes per version across two policies, and every one of them landed within about half a percentage point of the benchmark’s own wording. none of the prompt engineering habits carry over here, the tone and the structure and the think step by step do nothing to these models. what breaks them is leaving information out. there turned out to be three ways a normal, grammatical sentence does that.

none of those sentences are badly written. they’re what a person would say. every row has a control running the benchmark’s own complete instruction, which is the only way to know a low score isn’t just a task the policy can’t do at all.

the first row is the one that surprised me. it was supposed to be the clean case, and in 6 of 10 scenes the vague instruction made no difference at all. the policy has a favourite place to reach. under specified it goes there, which is right when the target happens to be sitting there and wrong when it isn’t, and naming the location overrides it completely. the ordering was consistent across all ten scenes, stove beats cabinet top beats cookie box, and once you have an ordering you have a prediction. two benchmark scenes use the same two locations with the roles swapped, same objects, same physics, fully instructed 8/8 in both. i wrote down which one should collapse before running them and got 7/8 and 2/8.

these blind spots also turned out to be per policy, and they invert. same two scenes, molmoact2 scores 83% and 21% while pi0.5 scores 12% and 79%, both at 100% when fully instructed. neither is worse than the other, they’re blind to different things, so there’s no fixed set of hard tasks to build against and you have to go find them per policy.

and one thing that looks like a language failure and isn’t. both policies run “open the top drawer and put the bowl inside” at 8/8 because it’s a task they were trained on. ask for the middle drawer instead, same opening skill, same insertion skill, both of which they have, and it’s 0/8. bottom drawer, 0/8. no phrasing fixes that.

before building anything i ran an untrained corrector as a floor. three arms from the same starting state, control repeating the vague instruction, corrector using whatever the VLM wrote, oracle just handing over the benchmark’s complete string. control lands at 3%, corrector at 19%, oracle at 100%. so the gap is recoverable by language and an untrained model gets a fifth of it, which felt like enough to justify training one.

then i read what the corrector was actually writing.

the vague instruction:   "pick up the black bowl and place it on the plate"
what the corrector said: "Pick up the black bowl from the drawer and place it on the plate."
what was true:           "pick up the black bowl on the wooden cabinet ..."

that one succeeded. the bowl is not in a drawer. another success named a milk carton in a scene with no milk carton. so a chunk of that 19% is specificity rather than correctness, because any concrete singular reference is enough to knock the policy off its default. it names the right object 5 times out of 15, and giving it the scene image doesn’t change that number. i asked Qwen3-VL-4B directly how many black bowls were in a picture containing two, at 256 and again at 512, and both times it said one. in fairness the second bowl is half out of frame in that view, but it scored 3/6 across the whole perception probe. so the bottleneck here is the corrector’s vision, not the policy’s language grounding.

so i ported the environment to CALVIN as well, same kind of benchmark, different sim, but the things you tell apart there are colours and lights and whether a drawer is open rather than two objects that look identical. perception went from 3/6 on LIBERO to 15/16, with colour naming at 8/8. it also inverted a tuning decision, LIBERO needed 512 and CALVIN gets worse at 512 because the model starts reading the table’s lower shelf as an open drawer.

both benchmarks then went through the same two gates. a task had to score at least 80% under the full instruction so i knew the policy could do it, and lose at least 50 points under the vague one so there was something to recover. of 46 candidate policy and task pairs on LIBERO, 12 survived. of 19 on CALVIN, 4.

the RL environment i wanted to build¶

the thing i was actually building towards is bigger than one corrector. imagine an environment where you register a policy, whatever it is, and the environment goes and finds the tasks that policy is worst at. not a fixed benchmark of hard tasks, because the per policy result above says that doesn’t exist, but a discovery process that produces a different list for every model you plug in. then an agent sits on top and its whole job is to learn how to talk to that specific policy, what it needs spelled out, what it assumes, where its priors will bite. bring your own VLA and the environment tells you how to drive it.

that’s a real product if it works, because everyone deploying one of these hits the same wall, and the failure list is different for every model and every scene.

the version i built is the smallest useful piece of that. one measurement decided its shape. i took the same episode and measured alignment twice, once at t=0 and once after driving the policy into committing on the wrong object. at the start it’s +0.867. after commitment it’s -0.643.

the sign flips. a late correction isn’t merely wasted, it pushes toward the wrong target, which is fatal for what i originally set out to build, because that design waits for something to visibly go wrong before it says anything and by then the lever has stopped working. so the environment applies its correction at t=0 of a fresh episode instead of mid rollout, which makes it a retry with a better instruction rather than a correction mid flight.

the rest of it is small. one turn, the corrector sees the vague instruction, the scene at the start and the final frame of the failed attempt, and writes one sentence under twenty words. the environment is the frozen policy running in sim across six servers, the reward is the benchmark’s own goal predicate on the retry, and there are no human labels anywhere in the loop.

the training set is six policy and task pairs. eval holds out whole scenes, and i split it so the strategy has to transfer rather than the answer, train on “turn on the stove and put the moka pot on it” and evaluate on “open the top drawer and put the bowl inside”, different scene, different object, different enabling verb. memorising the training answer cannot produce a correct held out one. i wrote the success bar down before spending anything, at least 50% on held out, and a kill condition of no better than the untrained baseline.

that’s the held out task, run with the full instruction. under the vague one the drawer never opens.

training the corrector didn’t beat prompting it¶

40 steps of GRPO on a 4B corrector over two policies in sim. the machinery was clean, all 40 steps trained, zero rollout errors, no truncation, and policy entropy dropped from 1.38 to 1.19, meaning its outputs got narrower and more repeatable, so it was converging on something.

everything below is measured on scenes the corrector never saw during training.

p = 1.00 on both rows, so neither of those drops means anything either. the trained corrector is simply not better than the untrained one.

it’s slightly worse than a null if you look at where the changes happened. the trained model went up on the tasks where the baseline scored zero and down on the ones where the baseline scored high, so it’s converging on a middle of the road answer rather than learning when to apply the fix.

the 50% bar came from a floor of roughly 12% i’d measured on Qwen3-VL-4B, but prime-rl’s registry forced me onto Qwen3.5-4B for the actual run, and that model’s untrained baseline is 33%. against the stale number, 27% reads as a fifteen point win. i only know it isn’t one because i ran the untrained model on the same servers the same day.

that second row separates whether the corrector applied the strategy from whether the policy managed to execute it, and training didn’t move it either. but when the step is supplied it works, and the trained model did produce the target behaviour once, on a held out scene with a verb it never saw in training, “Open the top drawer, then place the bowl inside.” so the capability is in there. what’s missing is the prior over when to use it. most of the successes on both sides are retry luck.

i’d written down 20 to 30% odds of clearing that bar beforehand, and a likely landing somewhere in 10 to 30%. it landed at 27%.

this wasn’t a small effect, it was no effect, and the untrained corrector i started from was already doing whatever there was to do. the way i read it now is that being sensitive to language and being steerable by language are not the same property. these policies are enormously sensitive to wording, but that sensitivity runs through priors and remembered phrasings rather than through anything like understanding. a sentence can knock a policy off course very easily and cannot reliably put it back, which is a bad property for something you want to use as a repair channel.

the failure data to train the corrector on is scarce here¶

these policies are not fragile in the way i expected going in. you can reword a task however you like and nothing happens. what they are sensitive to is missing content, and the failure is specific rather than general. leave out which of two identical objects you mean and success drops from 98% to 60%, name the wrong one and it goes to 3%. leave out a step that has to happen first and a task the policy does 8 out of 8 times goes to zero. and when you under specify, the policy falls back on a favourite place to reach, predictable enough that when two scenes used the same two spots with the roles swapped, i could say which one would collapse before running either.

what i don’t have is enough of these to train on. the gates left 12 pairs on LIBERO and 4 on CALVIN, six of which land in the training split, and six prompts is not enough for GRPO to learn a rule from, it’s enough for it to drift toward a safe average, which is what the per class numbers show. the list is short because most of what looks like a language failure is something else. swap a drawer and it’s a capability failure. ask about two identical bowls and it’s a perception failure, in the corrector, not the policy.

so the failures which are caused by language, fixable by language, and common enough to train on are a much smaller set than i assumed when i started.

i still think the environment is the right shape. you upload a policy, it goes and finds what that policy is specifically bad at, and a corrector trains on top to become a better triage agent for it. what it needs before that works is either a task generator that can produce these situations at scale instead of 16 of them verified by hand, or a corrector that can actually see the scene it’s describing, and probably both.

the whole thing cost about $100 in GPU, mostly incremental checkpoint training and evals.

the other thing holding it back is the policies themselves, which have to get better before any of this goes anywhere. language isn’t the modality i’d pick to experiment on right now. visual and input modality research looks more promising, and so does putting an llm agent on top of deterministic systems and having it route between them, rather than trying to talk a neural policy into behaving.

that’s what i’m on now, agents that can actually get physical tasks done. early signals from routing between deterministic robotic primitives look decent. will write that up when there’s something conclusive.