Interpretability and Explainability
Understanding why a model made a prediction, which matters when physicists and operators must trust it.
Why interpretability matters here
In fusion, a model's output may inform a control action or a design decision with real consequences. Physicists and operators need to understand and trust why a model predicts what it does, both to catch errors and to extract physical insight. An accurate black box is less useful than a slightly less accurate model whose reasoning can be checked.
Interpretable by design
The simplest route is to use models that are interpretable from the start: linear models with meaningful coefficients, small decision trees, or generalized additive models. On fusion's small datasets these often perform competitively, and their transparency is a real advantage.
Post-hoc explanation
- Feature-importance measures ranking which inputs matter
- Local attributions (such as SHAP values) explaining a single prediction
- Saliency maps highlighting which signal regions drove an output
- Partial-dependence plots showing how outputs vary with an input
Reading explanations carefully
Explanation methods are approximations and can mislead. A feature can appear important because it correlates with the true cause, not because it is the cause. Saliency maps can be unstable. Explanations are hypotheses about the model's reasoning, to be checked against physics, not accepted at face value.
From explanation to insight
At its best, interpretability turns a model into a source of physical understanding: if a disruption predictor consistently keys on a particular precursor, that is a physics lead worth following. At its most basic, it lets an operator sanity-check that a model is responding to the right signals and not to an artifact. Both uses are why interpretability is a first-class concern, not an afterthought.