Fault Tolerance and Failover
A fault-tolerant system keeps functioning despite component failures by detecting faults and switching to healthy resources without violating timing.
Continuing Through Failure
Fault tolerance is the ability to keep delivering correct service in the presence of faults. Rather than assuming components never fail, a fault-tolerant design assumes they will and arranges for the system to detect the failure and continue, either seamlessly or with graceful degradation. In real-time systems this must happen within timing constraints: a failover that takes too long is itself a failure.
Detect, Isolate, Recover
Fault handling proceeds in stages. First the fault must be detected, through self-tests, comparison of redundant channels, watchdogs, or out-of-range checks. Then it must be isolated so the faulty component does not corrupt the rest. Finally the system recovers, by switching to a standby, reconfiguring, or degrading to a reduced but safe mode. Each stage has a time bound, and the sum must fit within what the controlled process can tolerate.
Failover Styles
- Hot standby: a backup runs in parallel and takes over instantly on failure
- Warm standby: a backup is ready but must synchronize state before taking over
- Cold standby: a backup must start up, giving the slowest failover
- Graceful degradation: shed non-essential functions to preserve critical ones
State and Consistency
The hard part of failover is state. A backup that takes over must have, or quickly acquire, a consistent picture of the process so it does not command a discontinuity. Hot-standby designs keep the backup continuously updated so it can assume control within one or two control cycles. This tight synchronization is what makes failover fast enough for a fast control loop, where losing many cycles would be as dangerous as the original fault.
Fail-Safe as a Fallback
Not every function can be made to continue, and not every fault has a safe way to proceed. When continuation is impossible or unsafe, the correct behavior is to fall back to a defined safe state rather than to guess. A robust architecture combines redundancy for functions that must continue with fail-safe behavior for faults beyond what the redundancy can handle, so the system either keeps running correctly or stops safely, never runs incorrectly.