Reinforcement Learning for Heating and Current Drive
Reinforcement learning discovers policies that command heating and current-drive actuators to reach and hold target plasma states.
Control as sequential decision making
Heating and current-drive systems, such as neutral beams and radio-frequency waves, shape the plasma temperature and current profiles over time. Choosing how to command them to reach a target and hold it against disturbances is a sequential decision problem, which is the natural setting for reinforcement learning (RL).
The RL setup
An RL agent observes the plasma state, takes actions (actuator settings), and receives a reward measuring how close the state is to the target while penalizing constraint violations. Through trial and error, usually against a fast simulator, it learns a policy mapping states to actions that maximizes cumulative reward.
- State: profiles, stored energy, current, and their trends
- Actions: beam power, current-drive settings, fueling
- Reward: track target profile, respect limits, avoid instabilities
Training in simulation
Because exploratory actions on real hardware are risky and discharges are scarce, RL policies are trained in simulators built from reduced transport and equilibrium models. The fidelity of the simulator bounds the policy's real-world validity, so simulator error is the central concern and policies are transferred cautiously.
Robustness
A policy that exploits quirks of an imperfect simulator can fail on hardware. Domain randomization, which trains across a range of plausible plasma models, and conservative reward shaping improve robustness. Learned policies are typically wrapped by safety limits that override unsafe commands.
For heating and current-drive control of concepts like the Hyperion breeder, RL policies are developed against simulators ahead of construction. They command modeled plasmas because the machine is not built, and any policy would be validated on hardware before operational use, with no performance claimed in advance.