Field guide · robots in unstructured places
The Sim2Real Gap
A simulator is a hypothesis about the world. A policy trained inside it learns the hypothesis, including the parts that are wrong. The gap is everything the hypothesis leaves out. It is why a robot can succeed every time on screen and only most of the time on site.
Blue is always the simulatorOrange is always the real site
The same pick, 60 times
held missed the ±4 mm window slipped out during the lift
Every simulated attempt is identical, so 60 runs collapse into one line. On site, the same code on the same robot model spreads out. Calibration puts the camera’s estimate of the part a little off, sensing and slip add noise to every attempt, and an oily part sometimes slides out of the fingers. Run it again and the count changes. Move to another cell and the whole cloud shifts, because each cell carries its own calibration error.
Model: ±4 mm grasp window · random error σ 2.45 mm per attempt · cell bias drawn with σ 2.21 mm · slip when part friction is below 0.25. These are the defaults of Fig. 4.
The short version
- The gap is a mismatch between the worlds you simulated and the world you deploy into. Wrong numbers can be measured or randomized. Missing physics cannot be fixed by any setting.
- Simulators handle rigid-body motion well. Most of the gap sits in what they leave out unless you add it: actuator limits, sensor noise, calibration error, delay and wear.
- Small gaps compound. Millimetres stack against tolerances, milliseconds destabilize fast loops, and a 2% shortfall per step halves a 30-step task.
- Randomization buys coverage and costs sharpness. Adaptation, real data and forgiving hardware recover the rest.
- Repeatability belongs to the fleet, not the robot. Failures cluster by unit, site and day, and a success rate needs hundreds of trials before it means much.
Section 1
One world in training, another on site
Training in simulation means sampling worlds from a distribution someone wrote down: masses, friction values, camera models, object sets, lighting. The policy becomes good at that distribution. Deployment samples worlds from the real distribution, which nobody wrote down. The sim2real gap is the difference between the two, as the policy experiences it.
The mismatch comes in two kinds, and they need different fixes.
Parameter error
The simulator has the right physics with the wrong numbers. Friction is set to 0.8 where the part is oily and the real value is 0.3. Latency is zero where the real loop takes 40 ms. Fix it by measuring the real value, or by training across a range that includes it.
Model error
The simulator is missing the physics. Soil that compacts, a hose that whips, a dusty lens, a gearbox with backlash. No setting of the existing knobs reproduces it. Fix it by adding a model, learning one from real data, or making the robot indifferent to it.
Randomization buys coverage and costs sharpness
Expected success on site as the range widens
this policywith adaptation
Top: the friction a gripper actually meets across one site’s conditions. Bottom: where a policy trained on the shaded range succeeds. Training on one value works only near that value. Measuring the site’s average (system identification) lands between the conditions the site actually has. Widening the range covers them, but a policy that must work everywhere grips harder and moves more cautiously, so its success at any single value drops. A policy that infers friction from how the part responds recovers most of that loss.
Illustrative model. Site mix: 25% oily or wet parts (μ ≈ 0.30), 30% dusty (≈ 0.52), 45% clean and dry (≈ 0.78).
One slider understates the problem. A real robot has dozens of uncertain parameters, and the policy needs all of them inside its training ranges at the same time.
If each of 20 uncertain parameters lands inside its training range 95% of the time, all of them land inside together only 36% of the time.
Assumes the parameters vary independently. A policy that degrades gracefully just outside its ranges does better than this; one that fails sharply does worse.
This is why teams randomize generously, and why a policy that can estimate where it is in parameter space (Section 4) beats one that has to be robust everywhere at once.
Section 2
Where simulators fall short
Every robot runs the same loop. The world moves, sensors measure it, perception turns measurements into a state estimate, a policy chooses an action, and actuators push on the world. A simulator replaces each box with a model and each arrow with an instant hand-off. The gap lives in both places: in each model’s errors, and in the time each hand-off actually takes on hardware.
Select any gap to see what the simulator assumes, what reality does instead, how it shows up on site, and what teams do about it.
Twelve places the gap lives
The same gaps, sorted by how well a stock simulator captures them
The policy is the one box that runs identical code in both worlds. Only its inputs differ. Each Δt marks a hand-off that is instant in simulation and takes time on hardware. Bars show fidelity: two for approximate by default, one for absent unless you add it, none for hard to model at all.
Section 3
Why small gaps become big failures
Most individual gaps look small: a millimetre of calibration error, 40 milliseconds of delay, a 2% failure rate per step. They grow through three mechanisms.
Errors stack against a tolerance
A pick or an insertion succeeds when the gripper arrives within the task’s tolerance: the clearance, chamfer or compliance that absorbs error. Every link between camera and fingertip adds some error. In simulation most links are exact, so the budget is empty and success is 100%. On site, they all contribute.
Two kinds of error behave differently. Random error changes every attempt and widens the spread. Bias stays the same on a given cell and shifts the whole spread. It comes from calibration, kinematic error and deflection, and it differs from cell to cell. That is where deployment-to-deployment variation starts.
An error budget for one pick
Same every attempt, differs by cell bias
Different every attempt random
average cellworst 1 in 10 cellssimulator
Errors along the axis that matters, combined as root-sum-square: independent errors add in quadrature. Success is the share of attempts that land inside the window. Defaults are typical orders of magnitude for a camera about a metre from the work. Set them to your hardware.
Look at the smallest number
Arm repeatability, the figure on the datasheet, barely registers. Calibration and perception dominate. Fixtured automation never needed those to be good, because it taught one pose and repeated it. A messy tote needs every link right at once.
Delay destabilizes fast loops
In simulation the robot acts on the state it sees at the instant it sees it. On hardware the image is already tens of milliseconds old by the time the motors respond: exposure, readout, transfer, inference, bus. A controller tuned for speed in a zero-delay simulator has no margin left for that delay.
The same controller with and without delay
simulator, no delaysite, delay τpart position
An arm moves toward a part using camera feedback, commanding speed in proportion to the remaining distance (gain K, speed limit 0.5 m/s). With delay τ it acts on where the part appeared τ ago; the dashed outline on the site rail is what the camera last reported. Above π/2 the loop is unstable, and the speed limit turns the instability into a steady back-and-forth hunt. The lower gain tolerates three times the delay and settles more slowly.
Arms doing slow pick-and-place in position control tolerate delay well. Delay bites in fast or force-sensitive loops: visual servoing, contact and insertion, balancing, walking, anything near a person.
Long tasks multiply small failure rates
A task with many steps succeeds only if every step does. A simulator that is 2% optimistic per step is wildly optimistic over a 30-step task. Recovery changes the arithmetic: a failure the robot notices and retries costs time, not the task.
Task success against number of steps
simulatorsite, with recoverysite, no recovery
Each step succeeds with the probability set above. A failed step is noticed with the probability set above and retried up to the allowed number of times; an unnoticed failure ends the task.
Simulators rarely teach recovery from the failures that happen on site, because they do not produce those failures. That is one more reason to bring real failures back into simulation (Section 5).
Section 4
Closing the gap
There are four software moves and one hardware move. Narrow the gap by measuring reality and making the simulator match (system identification). Cover it by training across ranges wide enough to include reality (domain randomization). Infer it online, letting the policy estimate hidden parameters from what it observes (adaptation). Close the remainder with real data. And widen the tolerance with hardware that forgives error: chamfers, compliance, suction, force control.
Which technique addresses which gap
addresses it directlyhelps partly
Rule-of-thumb ratings for orientation, not benchmark results. The columns group the twelve gaps from Fig. 3.
The best-documented example measured the same policies in both worlds. Peng and colleagues trained a Fetch arm to push a puck to a target, entirely in simulation, then tested on the real arm.
One task, measured in both worlds
Success in simulation and on the real robot
simulationreal robot
Real-robot success when one randomization is removed (LSTM policy)
The policy with memory (an LSTM) can infer the dynamics from how the puck has responded so far, a built-in form of adaptation, and loses almost nothing on transfer. Without randomization, a policy that succeeds about half the time in simulation never succeeds on the robot. Removing just the action-timing or observation-noise randomization costs more than half of the real-world success.
Data: Peng, Andrychowicz, Zaremba & Abbeel, ICRA 2018, Tables II and III, ± as reported. Real-robot trials per policy: 28, 20, 12 and 10.
What is changing
GPU-parallel simulators make it cheap to train across thousands of randomized worlds at once. Isaac Lab and MuJoCo Playground are widely used, and in 2025 NVIDIA, Google DeepMind and Disney Research announced Newton, an open physics engine now hosted by the Linux Foundation. That makes randomization cheaper. It does not add physics a simulator is missing. Learned world models, trained on video, are starting to cover appearance and variety, with no guarantee that their contact physics is right.
Section 5
Working once versus working every time
Crossing the gap once, on one robot, on one day, is a demo. Deployment means the same result across units, sites, shifts and months. That is a separate problem, with its own failure modes.
Accuracy and repeatability are different numbers
Industrial standards separate two questions. Does the arm go where it was told, on average? That is accuracy. Does it go to the same place every time? That is repeatability.
Four ways to hit a target
commanded pose reached pose their mean repeatability radius RP
Accuracy (AP) is the offset of the mean from the command. Repeatability (RP) is the radius of the cloud around its own mean, computed here the ISO 9283 way from the dots drawn. Illustrative scale: 10 px = 1 mm.
Arms are built to be repeatable: a UR5e is rated at ±0.03 mm. Their absolute accuracy is far worse. An uncalibrated industrial arm is commonly off by around a millimetre and sometimes several, and careful calibration brings that down to a few tenths.
Why unstructured work is harder
Classical automation needs only repeatability. Teach a pose once, fixture the part, and the arm returns to it within hundredths of a millimetre for years. Unstructured work removes the fixture. Every target is computed from perception, so success depends on accuracy, which means every model in the loop being right at the same time. This is the sim2real gap showing up as a calibration problem, and it is why two identical cells can perform differently.
A fleet’s failures have structure
Run the same software on 12 cells at 3 sites for 30 days. The result is not one success rate. Failures cluster by unit, by site and day, and by time since the last calibration. Each pattern points to a different cause.
30 days of a simulated fleet
Each square is one unit’s failure rate for one day of 150 picks. Rows that stay dark point to a unit: hardware or calibration. Columns that darken across a site point to the environment: weather, light, a new batch of parts. Gradients that reset point to wear and recalibration. The fleet average hides all three.
Simulated data. Success on the logit scale = base + unit effect + site effect + site-day effect + occasional site shocks − wear × days since calibration, then sampled.
How many trials before a number means something
A success rate measured over a few trials is an interval, not a number. Many robot-learning results are reported over 10 to 50 real trials; the Peng et al. figures above used 10 to 28.
What a measured success rate actually tells you
Exact 95% confidence intervals (Clopper–Pearson). Ten out of ten is consistent with a true success rate as low as 69%.
A deployment loop that crosses the gap on purpose
Teams that deploy repeatably treat the gap as something to measure all the time, rather than something to close once. Each cycle crosses it twice: failures travel from site to simulation as logs and scans, and fixes travel back as versioned releases.
The deployment loop
- 1
Survey the site
Light levels across the day, surfaces, dust, temperature, network, the real mix of parts. Scan the cell. Turn it into a site profile and randomize around it.
- 2
Calibrate and self-test every start
Check hand–eye calibration against a fixed fiducial, home the joints, test the gripper, check sensor health. Refuse to start when residuals exceed the error budget.
- 3
Roll out in stages
Shadow mode first, where the policy proposes and the existing process acts. Then one canary cell, then the fleet. Keep the previous release one switch away.
- 4
Log every attempt, per unit
Outcome, failure type, latency percentiles, temperatures, calibration residuals. Alert on per-unit and per-site trends, not just the fleet average.
- 5
Rebuild failures in simulation
Classify each failure by gap source. Replay the logs, rebuild the scene, and add the condition to the randomization ranges and to the test suite.
- 6
Gate releases on sim and real
A release passes a simulated regression suite and a fixed set of real golden tasks, with every component versioned together.
Example: what has to be pinned together to reproduce a result
# deployment manifest (example values) release: pick-tote 2026.10.3 policy: weights 4be1…c09 · trained on sim 2.3.1 · randomization ranges r7 site: profile B · open canopy · 200–90,000 lux · dust high cell: B3 · gripper pads fitted 2026-10-02 calibration: hand–eye 2026-10-26 06:02 · residual 0.6 mm (budget 1.0 mm) firmware: drives 5.12.4 · camera 2.1.0 · gripper 1.8.0 gate: sim suite 4,800 episodes 97.6% · real golden tasks 120 / 120
A real golden-task result of 120 out of 120 still only shows the true rate is above about 97% (Fig. 11). The simulated suite catches regressions cheaply; the real tasks keep the simulator honest.
Make the simulator earn its vote
A simulator is useful as a release gate only if it ranks changes the way reality does. Kadian et al. measured this for nine navigation models: simulated and real rankings correlated at 0.18 until they removed a wall-sliding shortcut agents had been exploiting, then 0.844. Track that correlation for your own tasks, and treat a drop as a bug in the simulator.
Sources
- Tobin, J. et al. (2017). Domain randomization for transferring deep neural networks from simulation to the real world. IROS. arXiv:1703.06907
- Peng, X. B., Andrychowicz, M., Zaremba, W. & Abbeel, P. (2018). Sim-to-real transfer of robotic control with dynamics randomization. ICRA. arXiv:1710.06537
- Tan, J. et al. (2018). Sim-to-real: Learning agile locomotion for quadruped robots. RSS. arXiv:1804.10332
- Hwangbo, J. et al. (2019). Learning agile and dynamic motor skills for legged robots. Science Robotics 4(26). arXiv:1901.08652
- OpenAI et al. (2019). Solving Rubik’s Cube with a robot hand. openai.com
- Kadian, A. et al. (2020). Sim2Real predictivity: Does evaluation in simulation predict real-world performance? IEEE RA-L. arXiv:1912.06321
- Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V. & Hutter, M. (2020). Learning quadrupedal locomotion over challenging terrain. Science Robotics 5(47). arXiv:2010.11251
- Kumar, A., Fu, Z., Pathak, D. & Malik, J. (2021). RMA: Rapid motor adaptation for legged robots. RSS. arXiv:2107.04034
- Li, X. et al. (2024). Evaluating real-world robot manipulation policies in simulation (SIMPLER). CoRL. arXiv:2405.05941
- NASA/JPL (2009). Preparing for rover pivot test: the Spirit test bed for “Troy”. PIA12145
- NASA/JPL (2017). An algorithm helps protect Mars Curiosity’s wheels. jpl.nasa.gov
- Nubiola Batlle, A. (2014). Contribution to improving the accuracy of serial robots. PhD thesis, École de technologie supérieure. espace.etsmtl.ca
- Universal Robots. UR5e technical specifications (pose repeatability per ISO 9283). universal-robots.com
- ISO 9283:1998. Manipulating industrial robots: performance criteria and related test methods.
- NVIDIA (2025). Announcing Newton, an open-source physics engine for robotics simulation. developer.nvidia.com
Figures 1, 2, 4, 5, 6 and 10 run small illustrative models in your browser; their parameters are stated beside each figure. Figure 8 reproduces published results. Everything else is cited above. Source: github.com/ZeroTimeDrift/sim2real-gap.