After Imitation: Residual and Real-World RL as the Next Layer for Robot Policies

Imitation learning got robot foundation models out of the lab and onto diverse hardware. Vision-language-action (VLA) models can now map camera images and language instructions into motor commands across many tasks. But anyone who has watched a long-horizon rollout knows the soft spot: small execution errors stack up. A grasp that is a centimetre off becomes a failed insertion three seconds later. Demonstration data rarely covers those recovery states, so a purely imitation-trained policy has little idea what to do next.

In 2026, a growing body of work treats that gap as a separate learning problem. Instead of replacing the VLA, researchers add a next layer: residual reinforcement learning that corrects the frozen base policy, and real-world RL loops that selectively reuse deployment experience. This article explains those ideas with the numbers tied to the robots and tasks that produced them, and places them next to the sim-to-real toolkit that makes simulation-trained residuals practical.

Why imitation alone plateaus

A VLA is usually trained by behavioural cloning or flow matching on teleoperated demonstrations. That teaches the average successful behaviour in the dataset. It does not explicitly optimise for recovery when the robot drifts off the demonstration manifold. Physical Intelligence’s RECAP line of work, and related real-world RL systems, start from that observation: once a policy is deployed, human interventions and autonomous rollouts become a new data source for improving robustness and throughput [4][5].

Two complementary strategies have emerged:

1. Residual RL. Keep the VLA frozen. Train a smaller corrective policy that adds a delta to the VLA’s action. The residual can be trained in simulation if its observations transfer cleanly to the real world.
2. Iterative real-world RL. Deploy the policy, collect interventions and autonomous episodes, then update the policy itself using value estimates that favour efficient, low-intervention behaviour.

Both sit after imitation. They assume a capable base model already exists.

Residual RL in simulation that transfers zero-shot

Microsoft Research’s Object-Centric Residual RL for Zero-Shot Sim-to-Real VLA Enhancement, accepted at the 2026 Conference on Robot Learning (CoRL) in Austin, asks a sharp question: can a reinforcement learning policy trained only in simulation improve a real-world VLA without any real-world RL [1][2]?

The authors identify a familiar sim-to-real dilemma for residuals. Privileged-state residuals need distillation before they can run on the robot. Image-based residuals inherit the visual reality gap. Real-world RL is expensive and unsafe. Their answer is an object-centric residual: the corrective policy observes 6-DoF poses of task-relevant objects, the robot’s proprioceptive state, and the base VLA’s current action — not raw camera images [1][2].

The pipeline has three stages [1]:

1. Paired sim/real VLAs. Start from a VLA trained on real teleoperation. Replay the same teleoperation actions in simulation to train a simulation counterpart, so the two VLAs stay aligned in action space even when their pixels differ.
2. Residual training in simulation. Train the residual with pose noise injection and pose dropout so it tolerates imperfect real-world pose estimates.
3. Zero-shot deployment. At test time, swap in the real VLA and add the same residual to every action. No residual fine-tuning on the robot.

Results on a Franka Research 3

All real-robot numbers below come from five tabletop manipulation tasks on a Franka Research 3 (FR3), with 20 trials per task [1][2]. The base VLA in the paper is built on a GR00T-N1.5-style backbone [2]. Tasks: Cube Lift, Pick-and-Place, Stack Cube, Close Drawer, and Stand Cup Up.

The residual improves every task zero-shot. Average real-robot success rises from 42% to 76% without real-world reinforcement learning or residual-policy fine-tuning [1][2]. On successful episodes the authors also report shorter completion times, with reductions between about 9% and 22% depending on the task [1].

There is a second payoff. Residual-corrected rollouts can be aggregated and used to retrain the base VLA, turning a task-specific correction module into a self-improvement loop that needs no extra teleoperation [1][2]. That is residual RL as both an inference-time patch and a data generator.

Real-world RL with selective experience: Evo-RL

Simulation-trained residuals are one path. The other is to improve the policy from experience collected on the real robot. Evo-RL (Ye, Jin, Luo et al., arXiv:2610.13345, October 2026) is an iterative method that combines human interventions with advantage-guided experience selection [3]. This paper is an arXiv preprint and has not been peer-reviewed at the time of writing.

Evo-RL starts from a demonstration-trained policy built on the π0.5 architecture (a PaliGemma vision-language backbone plus a flow-matching action expert) [3]. During deployment, an operator intervenes when the policy stalls. The method then:

• Keeps only the human-controlled portions of successful intervention trajectories for corrective fine-tuning.
• Trains a vision-only value model with rewards that penalise both time and human takeovers.
• Estimates advantages and selects high-advantage segments from demonstrations and successful autonomous rollouts, rather than treating every frame equally.
• Redeploys the updated policy for the next round [3].

The authors contrast this with RECAP, which conditions policy training on a binary advantage label over accumulated experience [3][5]. Evo-RL uses estimated advantages to filter and reweight experience instead.

What the numbers are tied to

Evo-RL evaluates on two real platforms under a five-minute trial timeout [3]:

AgileX PiperX — high-precision screw insertion. Five screw–sleeve pairs sit on a tabletop. Either arm can insert a screw into a sleeve held by the other. Evaluation reports attempt success rate (ASR) and throughput across five sequential insertion units, with a success target of N = 10 per unit [3].

Under that protocol (Table II of the preprint) [3]:

• Demonstration-only supervised fine-tuning (SFT): 22.73% ASR.
• SFT plus one hour of human interventions: 78.72% ASR.
• RECAP: 57.78% ASR.
• First-round Evo-RL: 88.24% ASR and 60.61 successful stages per hour.
• Second-round Evo-RL: 92.16% ASR and 67.14 stages per hour.

The abstract’s headline — first-round Evo-RL improving ASR by 30.46 percentage points over RECAP — is exactly 88.24 − 57.78 under this PiperX insertion protocol [3]. It is not a claim about every robot or every task.

SO-101 — T-shirt folding. Five loosely piled T-shirts; units cover retrieve, spread, fold in half, fold sleeves, and fold hem to collar, with N = 20 per stage [3]. Complete Evo-RL raises ASR from 55.56% (SFT) to 76.67%, and throughput from 75.78 to 90.00 successful stages per hour [3].

A supporting artefact is RW-RL-Dataset: more than 1,000 hours of real interaction across four embodiments (AgiBot G2, Galaxea R1Lite, AgileX PiperX, and SO-101), with frame-level labels for whether a human or the policy was in control [3]. Code and checkpoints are released with the preprint [3].

How the two layers fit together

Residual sim RL and iterative real-world RL are not rivals. They attack different bottlenecks:

Industrial teams will often want both. Use simulation residuals (and the broader sim-to-real stack) to raise the floor cheaply. Use real-world RL loops to climb further on the contact-rich or long-tail failures that still escape the digital twin. The VLA remains the foundation; the RL layers are how that foundation stops being brittle.

CoRL 2026 itself, running 9–12 November in Austin, Texas, is a natural venue for this shift: the Object-Centric Residual RL paper appears on the CoRL 2026 programme, while real-world RL datasets and iterative methods continue to arrive as preprints that the community will pressure-test over the coming months [6][1][3].

What industrial teams should take away

For factory and warehouse pilots, the residual result is immediately actionable because it does not ask operators to run risky on-robot RL. If you can estimate object poses for the skills that matter — drawers, cups, cubes, pick-and-place targets — a sim-trained residual can raise success on a frozen GR00T-style VLA under the FR3 protocol above [1][2]. The self-improvement loop then turns corrected rollouts into new supervised data, which is attractive when teleoperators are the scarce resource.

Evo-RL’s message is different and complementary. On PiperX screw insertion, mixing interventions with advantage-guided reuse beat both interventions alone and RECAP under the authors’ ASR and throughput metrics [3]. On SO-101 folding, the same loop cut repeated attempts at the spreading stage while raising aggregate ASR [3]. That pattern matters for industrial AI robotics: spend human time on the failure modes the current policy actually hits, then let value estimates decide which autonomous frames deserve more weight.

Neither paper claims that residual or real-world RL replaces better base models, richer synthetic data, or careful system identification. They claim a narrower, useful thing: once you have a VLA, there is a measurable next layer.

Limits to keep in view

• Pose dependence. Object-centric residuals need reliable 6-DoF estimates of the right objects. Cluttered scenes and autonomous object selection remain open problems the Microsoft authors flag for future work [1].
• Human cost. Evo-RL still needs operators for interventions, even if targeted corrections beat more demonstrations under matched budgets [3].
• Scope of claims. The 42% → 76% residual result is five FR3 tasks, 20 trials each [1][2]. The 30.46-point Evo-RL gain over RECAP is PiperX screw insertion under the preprint’s ASR protocol [3]. Neither is a universal robot ranking.
• Preprint status. Evo-RL’s figures come from an arXiv preprint that has not completed peer review [3].

Bottom line

Imitation built the first useful generalist robot policies. The next layer is learning from mistakes — either by training a corrective residual in simulation that rides on a frozen VLA, or by iterating on real deployments with selective experience. In both cases the story is the same: keep the foundation model, and teach it how not to fail the last centimetre.

Related reading

• Vision-Language-Action Models: The New Foundation Layer for Robots
• Sim-to-Real in 2026: Closing the Gap Without Endless Real-World Data
• RoboColosseum and the Race to Measure Physical Intelligence in the Wild

Images: original illustrations generated for AndBeyond.Tech; no logos or real people depicted.