Watchdog and Heartbeat

The liveness layer that detects a silent, hung, or dead component and forces a safe action before the plant notices.

Detecting silence

The most treacherous fault is silence: a node that stops computing without saying so. Watchdogs and heartbeats make silence detectable. A heartbeat is a periodic I-am-alive message; a watchdog is a timer that forces a defined action if it is not petted in time. Together they bound how long a dead component can go unnoticed.

Heartbeat budget

The heartbeat interval and the watchdog timeout are chosen relative to the loop the component serves. A fast protection node must be declared dead within a few loop periods; a slow analytics node can tolerate seconds. The timeout must be shorter than the time for the un-controlled phenomenon to become dangerous.

class Watchdog:
    def __init__(self, timeout_ms, on_timeout):
        self.to, self.cb, self.last = timeout_ms, on_timeout, now_ms()
    def pet(self): self.last = now_ms()
    def check(self):
        if now_ms() - self.last > self.to:
            self.cb()          # force failover or safe action
            return False
        return True

Layered watchdogs

• Hardware watchdog on each real-time node forces a safe output on hang
• Supervisory watchdog tracks node heartbeats and initiates failover
• Plant watchdog confirms the supervisor itself is alive
• Independent protection hardware acts even if all software is lost

The bottom layer must not depend on software or the network, so that a total control-plane failure still leaves the machine able to reach a safe state. Heartbeat loss is the trigger for hot failover, and persistent watchdog trips are captured for postmortem.