Burst Buffers
Burst buffers are a fast flash tier between compute nodes and the parallel file system that absorbs bursty I/O and then drains it to disk.
Smoothing bursty I/O
Simulation I/O is bursty: long compute phases produce sudden floods of writes at checkpoints or output steps. A disk-based parallel file system is sized for sustained bandwidth, so these bursts either stall the application or force the file system to be over-provisioned. A burst buffer is a layer of fast solid-state storage placed between the compute nodes and the file system that absorbs a burst quickly, then drains it to the slower disk tier in the background while the simulation resumes computing.
Where the flash sits
Burst buffers come in two arrangements. In a node-local design, each compute node has its own SSD, giving bandwidth that scales perfectly with node count but staging data that is tied to that node. In a shared (remote) design, a pool of flash servers sits on the network between compute and the file system, usable by any node and better suited to shared-file patterns. Some systems offer both.
- Fast flash tier absorbs write bursts at high bandwidth.
- Data drains to the parallel file system asynchronously afterward.
- Node-local flash scales with nodes; shared flash suits shared files.
- Also caches read-heavy inputs to relieve the file system.
Uses beyond checkpoints
Besides absorbing checkpoint writes, burst buffers cache frequently read inputs so many ranks do not all hit the parallel file system, provide scratch space for temporary files in a workflow, and stage in-transit analysis data. Because the flash is a finite, shared resource, jobs typically request a capacity and duration through the scheduler, and data not drained is lost when the reservation ends.
In practice
A Hyperion checkpoint can be written to a burst buffer in seconds, letting the solver continue while the data trickles to the parallel file system in the background. This cuts the time the simulation spends blocked on I/O, which matters when checkpoints are frequent enough to protect against node failure on a long run.