Dead-Letter & Poison Messages

An event that repeatedly fails processing is quarantined to a dead-letter topic so it cannot stall a partition or block the machine.

One bad event must not stop the line

Because ordering is per-partition, an event that always fails handling would block every later event on its partition indefinitely. On a control plane that is unacceptable: a poison magnetics event could stall the whole breeder magnetics stream. Layer 4 bounds retries, then quarantines the offending event to a dead-letter topic and advances the offset.

Retry then quarantine

def process(e):
    for attempt in range(MAX_RETRY):
        try:
            handle(e); commit(e); return
        except Transient:
            backoff(attempt)         # transient: retry with backoff
        except Poison:
            break                    # deterministic failure: stop retrying
    deadletter.publish(e, reason=last_error, attempts=attempt+1)
    commit(e)                        # unblock the partition

Transient vs poison

• Transient (timeout, temporary unavailability): retried with exponential backoff.
• Poison (schema mismatch, invariant violation): not retried; sent straight to dead-letter.
• Ambiguous: retried a bounded number of times, then dead-lettered.

Safety-critical streams never silently drop

For command and safety streams, dead-lettering is itself an alarm: a command that cannot be processed triggers a fail-closed rejection and a loud alert, never a silent skip. A dead-lettered envelope or interlock event escalates immediately, because the safe interpretation of an unprocessable safety event is that the machine should hold or abort.

Draining the dead-letter

Dead-letter contents are inspected offline, root-caused, and, where a fix makes them processable, re-injected in order. Their lineage is preserved so a re-injected event carries its original identity for replay and audit. The dead-letter rate is a monitored SLO (see observability).