Trustworthy AI
Trust in an AI system rests on properties that must be engineered and evidenced: reliability, transparency, robustness, fairness, and accountability.
Trust Is Earned, Not Assumed
An AI system is trustworthy when there is evidence, not just hope, that it behaves as intended across the conditions it will face. Trust is not a feeling to be cultivated in users but a set of properties to be built in and demonstrated. Impressive demonstrations are not evidence of trustworthiness; systematic testing across realistic conditions is.
The Properties That Matter
- Reliability: it performs consistently across the intended range of inputs.
- Robustness: it degrades gracefully on unusual or adversarial inputs rather than failing silently.
- Transparency: its basis for a decision can be examined.
- Fairness: it does not encode harmful bias against groups.
- Accountability: responsibility for its outputs is clear, and errors can be traced and corrected.
Robustness and Distribution Shift
A model tested only on data like its training data can fail badly when the world shifts, encountering inputs unlike anything it learned from. Trustworthy systems are evaluated on out-of-distribution and adversarial cases, and, better still, know when an input is outside their competence and say so rather than guessing confidently.
Knowing What It Does Not Know
A central mark of a trustworthy system is calibrated uncertainty: its confidence tracks its actual accuracy, so a high-confidence answer is usually right and a low-confidence one flags itself for human review. A system that is confidently wrong is far more dangerous than one that is honestly uncertain, because it gives no warning.
In Scientific Use
AI applied to scientific problems must meet the same bar as any scientific tool: verified, validated in a stated regime, and honest about uncertainty and scope. A data-driven model is trusted only within its tested domain, and its outputs are treated as predictions to be checked, not facts, exactly as any model output is.