Incident Replay and Postmortem

Turning every fault into a reproducible record and a durable lesson - the loop that makes the stack more resilient over time.

Every fault is data

Resiliency is not static; the stack must get better after every incident. That requires an exact, time-aligned record of what the plant sensed, what the twin predicted, what the controller decided, and what actually happened. Incident replay reconstructs the event; postmortem extracts the lesson and feeds it back into FMEA, models, and drills.

Deterministic replay

Because the fast loops are deterministic, the same inputs replayed through the same controller produce the same outputs. Capturing every input at full rate lets engineers re-run the incident offline, test whether a proposed fix would have caught it, and validate the fix before it touches hardware. The unambiguous state trace makes the timeline exact.

def replay(record, controller):
    outputs = []
    for frame in record.frames:            # time-ordered captured inputs
        u = controller.step(frame.state)   # deterministic re-execution
        outputs.append((frame.t, u))
    return outputs   # compare against record.actual to localize the fault

The postmortem loop

• Reconstruct the timeline from the state trace and telemetry
• Localize root cause with replay and FDI signatures
• Update the FMEA if the mode was un-enumerated or mis-scored
• Add or retrain a detector so the mode is caught earlier next time
• Add a fault-injection case to the drill suite

Postmortems are blameless and evidence-driven; the output is a change to the system, not to the operator. Lessons validated on one unit propagate across the fleet. Pre-FOAK, incidents are simulated fault injections against the twin, which builds the replay and postmortem discipline before any hardware exists.