Model Drift and Monitoring
Deployed models degrade as the world changes; monitoring detects data and concept drift so models can be retrained in time.
Models decay after deployment
A model is trained on a snapshot of the world, but the world keeps moving. Over time the live data drifts away from the training distribution and accuracy silently erodes. Monitoring is the practice of watching a deployed model and its inputs to catch this decay before it causes harm, and it is a defining responsibility of MLOps.
Kinds of drift
- Data drift (covariate shift): the input distribution changes while the input-output relationship holds
- Concept drift: the relationship between inputs and target changes, so past patterns no longer apply
- Label shift: the class base rates change
- Upstream data issues: schema changes, broken pipelines, or new missing values that masquerade as drift
Detecting drift
Ground-truth labels often arrive late or never, so monitoring leans on proxies. Input drift is measured by comparing feature distributions against a training reference using tests like the Kolmogorov-Smirnov test, population stability index, or divergence measures. Prediction drift watches the distribution of model outputs. When labels do arrive, performance metrics are tracked directly, and confidence or calibration degradation is an early warning.
Responding
Detection is only useful with a response plan: alerting on threshold breaches, triggering retraining on fresh data, rolling back to a prior model, or falling back to a safe default. Retraining cadence can be scheduled or drift-triggered. Because retraining introduces its own risk, changes are validated and often rolled out gradually with shadow deployments or canaries, closing the loop from monitoring back to training in a maintained ML system.
Drift monitoring also intersects with anomaly detection, which flags individual abnormal inputs rather than distributional shifts.