Fifty Failures a Day
What it takes for a robot to leave the simulator and get a job
Overnight, in a server rack somewhere, a robot folds ten thousand towels. By morning, it folds a towel more neatly than you do.
Then someone hands the real robot a real towel, and it fails.
The simulated towel was a mesh of triangles obeying tidy equations. The real one is terry cloth, which bunches and clings and is heavy because it’s damp. So the robot’s fingers close on air, or close too hard, until the towel collapses into a heap.
This is the sim-to-real gap. In the lab, researchers celebrate a 95% success rate, but if a warehouse runs that same robot a thousand times a day they will see fifty failures, each time requiring a person to walk over to sort it out. They want to see 99.9% reliability.
The reason this is hard starts with data. Similar to language models, robotics models learn by training on an enormous number of examples, only those examples barely exist. A language model can read the internet; but there is no internet of robot actions, no trillion-token record of what it feels like to pour a coffee or peel apart two sheets of cardboard.
Simulation is the obvious way around this limitation - a simulator is infinite, free and safe. Jim Fan at NVIDIA put it as: you don’t train one robot, you train ten thousand in parallel, and the real world is just the ten-thousand-and-first. But those simulations fail too regularly when robots come into contact with the real world.
This is important because, despite over $30bn invested in robotics over the last 18 months, the sim-to-real gap remains a key barrier to the rapid adoption of robots in industry (let alone in the home).
So the question is simple - how do you make what a robot learns in simulation survive contact with reality, at warehouse odds rather than research odds?
Contact, it turns out, is the operative word.
The anatomy of the sim-to-real gap
The sim-to-real gap is really three gaps:
The perception gap: the simulated camera sees a cleaner, more orderly world than the real one.
The actuation gap: the difference between what the policy asks for and what real motors deliver with their lag, friction, and wear.
The contact dynamics gap: simulated physics doesn’t move the way matter moves.
Each of these needs to be solved to achieve the reliability needed, and each is at a different stage of progress.
Where transfer stands today
Perception
Perception has made dramatic progress in the last decade. The breakthrough came in 2017 from a team at OpenAI who tried the opposite of what everyone assumed. Instead of labouring to make a simulation look realistic, they made it look deranged: thousands of examples of surface texture, lighting, and camera angles, none of them remotely plausible. With that diversity of images, the network can’t rely on appearances, so it learns the underlying shapes instead. Trained entirely on nonsense, it located real objects to within 1.5 centimetres.
What began as a lab experiment has been industrialised. NVIDIA’s tooling now trains working object detectors from synthetic images alone and Gaussian splatting can rebuild a customer’s actual warehouse from video, so the simulator looks like the site. Vision also had one advantage no other part of robotics enjoys: an internet’s worth of real images to pre-train on.
None of this makes perception finished. Transparent objects, mirrors, and finding the exact position of an object in a jumbled bin still cause grief. But the camera is approaching generality.
Actuation
This isn’t discussed enough. A policy doesn’t move a robot; it asks motors to, and real motors are variable. There are friction, elasticity, and processing delays that are hard to model.
A fix was demonstrated in Zürich in 2019. Jemin Hwangbo and his colleagues at ETH ran the motors of their ANYmal quadruped through four minutes of exercises — sine-wave leg movements at varying amplitudes, plus the occasional manual shove — and logged more than a million samples of what the motors actually did in response to the commands they were given. They trained a tiny neural network to imitate those quirks and put it inside the simulator, so the simulated robot’s joints lagged and resisted like the real ones. Policies trained against it transferred straight to hardware; recovery from a fall worked on the very first attempt.
One founder we spent time with recently described onboarding a new customer as weeks of sensor and actuator calibration before any training happens, and recalled a customer’s calibration collapsing over something as trivial as a changed lens coating on a sensor.
This unsexy calibration will be critical to ongoing uptime and fleet maintenance in future.
Contact Dynamics
In 1988, the roboticist Hans Moravec pointed out an uncomfortable pattern: it was comparatively easy to make computers perform like adults at board games and intelligence tests, and nearly impossible to give them the skills of a one-year-old at perception and movement. Nearly four decades on, his paradox remains partially true. Walking, which looks hard, crossed over years ago — by 2022, a quadruped trained entirely in simulation could hike an Alpine trail at close to human pace, reading snow that looked like solid footing but wasn’t. Manipulating objects, which looks easy, is where the field still struggles.
A physics engine works by taking tiny steps forward in time and solving equations at each step: things speed up gradually, slow down gradually, bend gradually. Contact dynamics refuses to play along. Push a box across a table and friction holds it, holds it, holds it — then it breaks loose all at once.
Then wrap five fingertips around a jam jar. Each one is sticking, slipping, rolling or letting go from moment to moment; each combination obeys different equations, and the simulator has to work out which combination is true each second.
So every physics engine cheats. Solving the contact equations exactly is intractable. A classic 1991 result showed the core problem is NP-complete, so engines relax it into something a computer can solve. MuJoCo, the workhorse of robotics research, softens contact and lets objects sink fractionally into each other, so every surface becomes a very stiff spring and the engine has a formula to evaluate instead of a combination to search. Look closely at a MuJoCo simulation and you’ll see the robot’s fingers sitting slightly inside the jar they’re gripping.
These shortcuts are invisible to the human eye. But a reinforcement learning agent running millions of attempts will find them, learning to lean into the penetration or exploit the simplified friction. The tricks that work in simulation are exactly what fails in reality.
This is even harder for soft objects - fabric, food, cardboard. No engine simulates these faithfully yet, which is why the towel from the top of this essay still collapses.
Startups are trying to solve the problem with their own hardware. If you can’t trust a simulator to get contact right, you can build a robot that feels contact for itself. 1X, the company behind the NEO home robot, published a detailed account of its hands last week. A conventional robot hand has a high gear ratio that means each joint is stiff and doesn’t react to force — it drives to the position it was commanded and either crushes the object or slips. NEO’s fingers use ratios low enough to be backdrivable — push on a finger and the motor turns — so the hand gives way to what it touches, and the joints that apply force can also measure it. Tactile skin on the fingertips reads force, contact location, and shear.
The result is a hand that reacts to real contact as it happens, instead of trusting a simulator to have predicted it. It’s the same forgiveness that allowed walking to cross over first: a foot that gives under load doesn’t need a perfect model of the ground. It doesn’t make the gap disappear, you still have to teach the hand what to do, but it moves the hardest part out of the simulator and into the hand.
How the labs attack it
Brute Force
More parallel simulations, more randomisation, trust scale to do its work. This works when the simulator is only wrong in degree — a physics engine gets rigid bodies qualitatively right but doesn’t know the exact friction of the floor or weight of the object. Those can be randomised until the real values sit comfortably within the range simulated.
The randomisation can even be aimed: in 2024, researchers had GPT-4 write the training rewards and choose the randomisation ranges for a robot dog learning to balance on a yoga ball. No simulator reproduces a yoga ball’s wobble exactly, but randomise the bounce across a wide enough range and the real ball sits inside it. It transferred with, in Jim Fan’s words, “No fine-tuning. Just works.”
Unfortunately, the damp towel can’t be randomised to success — the fast engines can’t manage the physics, and the soft-body solvers are too slow to run millions of times over.
Learning physics from video
Several labs are betting that with enough video and action feedback, a model can absorb physics the way a language model absorbs grammar, producing a “world model” you can act inside. NVIDIA’s Cosmos, Odyssey’s 2-Max, and Wayve’s GAIA-3 are all emerging contenders. If it works, these world models would allow a much wider diversity of simulated scenes. Jim Fan describes handcrafted engines as “Simulation 1.0” and expects learned world models, “Sim 2.0”, to take over as compute scales.
Calibration
This is the mirror image of brute force. Randomisation covers uncertainty: prepare the robot for every friction the floor could plausibly have. Calibration removes it: measure the floor, the motor, the shelf.
A University of Washington system called ASID has the robot nudge an unfamiliar object, watch how it moves, and fit its weight and friction from the response — the way you heft an unfamiliar jar before trusting your grip — then re-plan inside a simulator corrected for that object. This is perfect for when the robot will stay in the same environment consistently, which is most commercial deployments today.
Calibration also scales up from the object to the site. RialTo, out of MIT, scans a room with a phone, reconstructs it as a digital twin, and trains against the customer’s actual shelves and drawers rather than a generic warehouse; the resulting policies are roughly 67% more robust than ones trained from demonstrations alone.
Brute force and calibration combine well — the Zürich team measured their motors precisely, then randomised whatever the measurements missed.
Collecting Data
Accumulate real-world data via everything from instrumented gloves to robots learning on the job. Physical Intelligence has taken this furthest. Its latest model learns by reinforcement from its own deployments — folding laundry, making espresso, assembling boxes — with each shift's successes and failures folded into the next round of training; the company reports this "more than doubles the throughput on some of the hardest tasks."²⁰ So when a shipment of cardboard arrives with perforations that make the sheets stick together, a deployed robot learns to handle it.
Why this matters
The customer is not buying a SOTA model that works in a demo video. They are buying a job done reliably - cases moved per hour and a clear plan for what to do when a gripper jams and the line stops at 2am. One global carmaker told us they’d need to see a 95% success rate from a simulation vendor on the first real-world deployment, with manual tuning to improve it from there.
This is what a real trial looks like. For most of last year, a Figure humanoid stood at a fixture in the body shop of BMW’s plant in Spartanburg, South Carolina, feeding sheet metal toward the welding line — the same task, hour after hour, through more than 30,000 X3s by the company’s count. In November BMW retired the fleet, and this June it announced the successor project: a newer robot, a harder job, this time in logistics. BMW frames this as a successful pilot, not yet a proven business case. But a demanding customer ran the robot for most of a year, then committed to something harder. Pilots should be focused on a tightly scoped task with a decision gate at the end.
The startups in deployment today prioritise reliability. They ship something narrow and boring, constrain the objects and the layout until it’s reliable, then widen the deployment once they’ve earned trust.
Evan Beard of Standard Bots described it succinctly: “We get paid to learn. We get better, faster because we are forced to interact with reality.” Chef Robotics assembles meals in food-production facilities, scooping and portioning food that clumps and smears, never sitting in the scoop the same way twice. That is the deformable, high-variance contact the fast simulators can't reproduce. By its own count it has passed 100 million servings in production, doubling in under a year. With each new kitchen and set of ingredients they build a real-world data advantage over competitors.
Test, fail, update, repeat
The sim-to-real gap won’t be fixed in one stroke. A policy is released, reality finds the cases it missed, and the next version has to fix them without breaking anything else. Progress is a loop: collect, train, test, deploy and fold the failures back in.
Testing in simulation is much quicker and cheaper than physical testing, which can take weeks to organise and cost thousands. A simulated test costs nothing and repeats the same every time - the same box, the same wet morning - allowing teams to test new models easily. Every failure the fleet has ever logged can be kept as a permanent test, the robot equivalent of software’s regression suite, so releasing an update to improve pallet handling won’t accidentally break tote handling.
Frustratingly, while failures of geometry and layout replay faithfully, failures of contact replay only approximately. The sticky cardboard can be reconstructed down to the millimetre, but we still don’t know if it will stick the same way the second time around.
Ultimately, physical deployment remains the ground truth and is necessary to catch the failures no one could predict in a simulation. A successful team will do the vast majority of testing in simulation to catch obvious errors, use physical testing and real-world deployments to catch unexpected failures, then rebuild those failures into simulated tests to evaluate the next iteration against.
Where to from here?
None of this will be won by the best demo, but by the teams already on a real floor, turning failures into training data that compounds their advantage over time. If you’re building the narrow, deployed version of this, or the tooling that lets everyone else test and improve theirs, we’d love to hear from you: jackie@airtree.vc.