Parallel File Systems
Parallel file systems such as Lustre and GPFS spread files across many storage servers so thousands of clients can read and write at once.
Storage that scales
A single disk or server cannot feed a supercomputer. A parallel file system distributes data across many storage servers and devices, presenting a single namespace to clients while delivering aggregate bandwidth that scales with the number of servers. The two most common in HPC are Lustre and GPFS (IBM Storage Scale). They let thousands of compute nodes share files at high combined throughput.
Separating data from metadata
A defining design choice is splitting data from metadata. In Lustre, file contents live on Object Storage Targets served by Object Storage Servers, while the directory structure, names, and permissions live on Metadata Targets served by Metadata Servers. A client asks the metadata service where a file's pieces are, then talks directly to the data servers for the bytes. This keeps bulk data transfer off the metadata path so both scale independently.
- Data is striped across many servers for aggregate bandwidth.
- Metadata is handled separately so lookups do not throttle transfers.
- One global namespace spans all compute nodes.
- Metadata operations (many small files, stat storms) are the usual bottleneck.
The metadata pitfall
Parallel file systems deliver huge streaming bandwidth but handle floods of small operations poorly. A job that opens, stats, or creates millions of tiny files hammers the metadata service and slows the whole system for everyone. The guidance is to use few large files with parallel I/O rather than many small ones, and to avoid per-process output and repeated stat calls in hot loops.
In practice
A Hyperion campaign writes a handful of large shared parallel-HDF5 files rather than one file per rank per step, and reads shared input tables via broadcast rather than having every rank stat and open them. This keeps the metadata service healthy and lets the run use the file system's streaming bandwidth as intended.