Sim-to-Real Transfer

Policies trained in simulation must survive the reality gap; domain randomization and adaptation help them cross it.

Why train in simulation

Real-world RL is slow, costly, and often unsafe. Simulation is fast, parallelizable, and forgiving of exploratory mistakes. The catch is the reality gap: a policy that excels in a simulator can fail on hardware because the simulator's physics, sensors, and latencies differ from reality.

Domain randomization

The most influential technique is domain randomization: rather than model reality precisely, randomize simulator parameters (masses, friction, textures, sensor noise, delays) across a wide range during training. A policy forced to perform under this variety learns robust behavior and treats reality as just one more variation it has already seen. Randomizing visuals likewise lets vision policies transfer from rendered to real images.

• Dynamics randomization: physical parameters like mass, friction, actuator gains
• Visual randomization: lighting, textures, camera pose for perception
• Latency and noise: control delay and sensor error, often overlooked but decisive

Closing the gap further

Domain randomization can be made adaptive: automatic domain randomization gradually widens ranges as the policy improves. System identification tunes the simulator to match measured real data. Domain adaptation aligns simulated and real feature distributions. Online adaptation methods infer the true parameters at deployment from a short interaction and adjust the policy accordingly, connecting sim-to-real to meta-RL.

# Domain randomization step
params = {
  'friction': uniform(0.5, 1.5),
  'mass':     uniform(0.8, 1.2) * nominal_mass,
  'delay_ms': uniform(0, 40),
}
env.set_params(params)   # new sample every episode

Practical stance

Sim-to-real is now the default recipe for learned robotic control, from in-hand manipulation to legged locomotion. The reliable pattern is: randomize broadly, keep a conservative safety layer on hardware, and validate on the real system rather than trusting simulated performance.