Robotics
Simulation, sim-to-real and how robots learn now
Training a controller in a physics simulator and transferring it to hardware became practical, and it changed how the field works.
For decades, robot controllers were designed analytically: model the system, derive a control law, tune it on hardware. That approach still underpins safety-critical control, and much of what is new is learned rather than designed.
The data problem
Learning a control policy requires enormous experience. Reinforcement learning methods may need millions of trials.
Collecting that on real hardware is impractical: robots break, batteries deplete, and a million trials at real-time speed takes years.
Simulation solves this. A physics engine can run many environments in parallel, faster than real time, without wearing anything out.
Modern setups run thousands of simulated robots simultaneously on a single accelerator, accumulating years of experience in hours.
The reality gap
The obstacle that made this impractical for a long time.
A simulator is an approximation. Friction, contact dynamics, actuator response, sensor noise and material deformation are all modelled imperfectly.
A policy trained to exploit the simulator's specific inaccuracies fails immediately on hardware — and reinforcement learning is extremely good at finding exploits, which is precisely the problem.
Domain randomisation
The technique that largely solved it, and it is elegantly simple.
Rather than trying to make the simulator perfectly accurate, randomise its parameters during training: vary friction, mass, actuator strength, latency, sensor noise and visual appearance across a wide range.
The policy cannot rely on any specific value, so it learns a strategy robust across the whole distribution.
If the real system's parameters fall within that distribution, the policy works on it — treating reality as one more sample rather than as a special case.
This is why legged locomotion transferred successfully: gait control is exactly the kind of problem where robustness to parameter variation is what you want anyway.
System identification
The complementary approach.
Measure the real system carefully and tune the simulator to match it, narrowing the gap rather than papering over it with randomisation.
In practice the two are combined: identify what you can, randomise around it.
Where transfer works well
Locomotion. The standout success. Quadruped and biped gaits trained entirely in simulation now walk on hardware over terrain they never saw, and this is standard practice rather than a research result.
Flight control. Well-modelled dynamics, minimal contact.
Coarse manipulation. Reaching, pushing, simple grasping of rigid objects.
Where it transfers poorly
Contact-rich manipulation. Inserting a peg, tightening a screw, handling a deformable object.
Contact dynamics are the hardest thing to simulate accurately. Friction at the microscale, surface deformation and stick-slip behaviour are approximated crudely by every engine.
Deformable objects. Cloth, cable, food, soft tissue. Simulating these accurately is computationally expensive and still inaccurate.
This is a substantial part of why manipulation lags locomotion so badly.
Anything involving fluids, for the same reasons at greater cost.
Learning from demonstration
The parallel approach that has gained ground where simulation struggles.
Rather than learning in simulation, collect demonstrations from human teleoperation and train a policy to imitate them.
This works for contact-rich tasks precisely because it sidesteps simulating contact — the demonstrations contain real contact dynamics implicitly.
The cost is data collection: every demonstration requires a person operating the robot, which does not parallelise the way simulation does.
Large-scale demonstration datasets, shared across institutions, are an active effort to address this, on the hypothesis that a policy trained across many robots and tasks generalises better than one trained on a single setup.
What this changed about the field
Iteration speed. A behaviour can be trained overnight and tested in the morning.
Who can participate. Simulation is cheap; a research group without expensive hardware can contribute.
What is attempted. Behaviours too dangerous or too time-consuming to learn on hardware — recovering from a fall, traversing unstable terrain — became routine to train.
The remaining caution
A policy trained in simulation has no guarantees. It works because it was tested, not because it was proven.
For safety-critical applications, that matters: certification frameworks are built around analysis and verification, and a learned policy resists both.
Which is why learned control appears first in applications where failure is inconvenient rather than dangerous, and why hybrid architectures — a learned policy inside a verified safety envelope — are the direction for anything where it matters.
