Message Queues and Brokers
A message broker decouples producers from consumers by durably buffering and routing records, absorbing bursts and surviving failures.
A shock absorber between systems
When one system produces data and another consumes it, connecting them directly is fragile: if the consumer is slow or down, the producer stalls or loses data. A message broker sits between them, accepting records from producers and holding them durably until consumers are ready. This decoupling lets each side operate at its own pace and fail independently.
Queues versus logs
Two broker models dominate. A queue delivers each message to one consumer and removes it once acknowledged, good for distributing tasks. A log (append-only) keeps messages in order and lets many consumers read at their own offsets, replaying history as needed. Log-based brokers underpin most modern streaming.
- Topic/stream: a named channel messages are published to.
- Partition: a subdivision of a topic enabling parallelism and order within it.
- Offset: a consumer's read position in a partition.
- Consumer group: a set of consumers sharing the work of a topic.
Delivery and durability
Brokers persist messages to disk and often replicate them across nodes so a single failure loses nothing. They offer delivery guarantees, at-least-once being common, and let consumers commit offsets so they resume exactly where they left off after a restart. See event sourcing and logs.
Backpressure
When consumers cannot keep up, a broker's buffer grows. Well-designed systems apply backpressure, signaling producers to slow, or rely on retention limits that expire old data, rather than exhausting memory. Sizing retention and throughput to the expected burst is a key design task.
In a fusion program
A broker fits between high-rate diagnostic acquisition and the systems that store, monitor, and analyze the data, absorbing per-pulse bursts and letting the live dashboard and the archival writer consume the same stream independently.