Sim-to-Real Transfer
Policies trained in simulation must bridge the reality gap to work on physical systems, using randomization and robust training.
Train cheap, deploy real
Sim-to-real transfer is the practice of training a policy in a fast, safe simulator and then deploying it on a real physical system. Simulation makes interaction cheap and exploration harmless, but a policy that excels in simulation can fail on hardware because the simulator never perfectly matches reality — the reality gap.
Why the gap exists
- Unmodeled physics: friction, delays, flex, and noise the simulator approximates or omits.
- Sensor and actuator differences: real signals are noisier and lag more than idealized ones.
- Parameter mismatch: masses, gains, and constants differ from their simulated values.
- Overfitting: the policy may exploit quirks of the simulator that do not exist in reality.
Domain randomization
The leading remedy is domain randomization: vary the simulator's parameters — dynamics, delays, noise, appearance — across a wide range during training. A policy that performs well across many randomized variants treats the real world as just one more variant, and so transfers more robustly. It trades peak simulated performance for real-world robustness.
Other techniques
System identification tunes the simulator to match measured hardware behavior, shrinking the gap directly. Domain adaptation aligns simulated and real observations. Robust and adversarial training prepares the policy for worst-case disturbances. Often a policy is fine-tuned briefly on the real system after simulation training, combining cheap bulk learning with a small dose of real data.
Relevance to physical plants
For controlling costly or not-yet-built systems, sim-to-real is the only viable path: exploration on the real device is unacceptable. Kronos machines such as the Hyperion breeder exist today only as design and simulation studies, so any control policy would necessarily be developed and validated in simulation, with the reality gap treated as a first-class engineering problem before any hardware exists.