Fairness and Bias in Machine Learning
Fairness in ML studies how models can encode or amplify social bias, and the trade-offs among competing fairness definitions.
Where bias comes from
Machine learning models learn from historical data, and that data reflects the world as it was, including its inequities. A model trained to reproduce past decisions can encode and even amplify discrimination against groups defined by protected attributes such as race, sex, or age. Bias enters through skewed sampling, biased labels, proxy features that stand in for protected attributes, and feedback loops where model decisions shape future data.
Competing definitions of fairness
- Demographic parity: positive prediction rates are equal across groups
- Equal opportunity: true-positive rates are equal across groups
- Equalized odds: both true-positive and false-positive rates are equal
- Calibration within groups: predicted probabilities mean the same thing in each group
- Individual fairness: similar individuals receive similar predictions
Impossibility results
These definitions often cannot be satisfied at once. A well-known result shows that when base rates differ across groups, a model generally cannot be both calibrated and equalized in error rates. Fairness is therefore not a single switch but a value-laden choice among trade-offs that depends on context and on who bears the cost of each error, a decision for stakeholders, not the algorithm alone.
Mitigation across the pipeline
Interventions apply at three stages: pre-processing reweights or transforms the data to reduce bias; in-processing adds fairness constraints or penalties to the training objective; post-processing adjusts thresholds per group after training. Each has trade-offs with accuracy and legality. Auditing relies on interpretability and disaggregated evaluation, reporting metrics per group rather than only in aggregate, so harms to small groups are not hidden by good averages.
Fairness work is inseparable from documentation and governance, tying it to model cards and the monitoring practices of MLOps.