Model Interpretability
Interpretability methods explain why a model made a prediction, supporting trust, debugging, and accountability.
Opening the box
As models grow more accurate they often grow more opaque, yet many settings demand explanations: to debug failures, to satisfy regulators, to catch spurious shortcuts, and to earn user trust. Interpretability is the toolkit for answering why a model predicted what it did, at the level of a single prediction (local) or the model as a whole (global).
Intrinsic versus post-hoc
Some models are interpretable by construction: linear models expose coefficients, shallow decision trees expose rules, and generalized additive models show each feature contribution as a curve. When accuracy demands a complex model, post-hoc methods explain it after the fact, treating it as a black box and probing its behavior. There is often, though not always, a tension between raw accuracy and built-in transparency.
The main post-hoc tools
- SHAP: attribute a prediction to features using a game-theoretic fair-share rule
- LIME: fit a simple local model around one prediction
- Saliency maps: highlight input regions a neural net relies on
- Permutation importance: measure global feature importance by shuffling
- Counterfactuals: the smallest input change that flips the outcome
Explanations are not ground truth
An explanation is a model of a model, and it can be wrong or misleading. Different methods can disagree; some are unstable to small input changes; and a plausible-looking explanation does not prove the model reasons that way. Explanations should be validated (do they predict the effect of interventions?) and used to generate hypotheses to test, not accepted as certified accounts. Interpretability also underpins fairness auditing, where the question is whether a model relies on protected attributes.