RDMA and InfiniBand
Remote direct memory access moves data between nodes' memories without CPU involvement; InfiniBand is the interconnect that popularized it in HPC.
Direct memory over the network
Remote Direct Memory Access (RDMA) lets a network adapter read from or write to a remote node's memory directly, bypassing the remote CPU and, on the local side, avoiding extra data copies through the operating system. This zero-copy, kernel-bypass path gives very low latency (single-digit microseconds) and high bandwidth, and it frees CPU cycles that a copy-based network stack would consume.
InfiniBand and its relatives
InfiniBand is the interconnect that made RDMA standard in HPC clusters. It provides high per-link bandwidth, low latency, and hardware support for RDMA verbs and collective offload. RoCE (RDMA over Converged Ethernet) brings the same RDMA semantics to Ethernet fabrics. Applications rarely use these directly; MPI implementations sit on top of the verbs interface and expose the performance through ordinary send, receive, and one-sided calls.
- Zero-copy transfers skip intermediate buffering.
- Kernel bypass avoids OS overhead on the data path.
- Remote CPU is not interrupted for one-sided transfers.
- MPI and PGAS runtimes are built on RDMA verbs underneath.
Topology matters
Clusters wire these links into topologies such as fat-tree and dragonfly that aim to provide high bisection bandwidth so many pairs of nodes can communicate at once without contention. The topology strongly affects collective performance, especially all-to-all, and job schedulers try to place a job's ranks on topologically nearby nodes to shorten paths and reduce contention with other jobs.
In practice
Large Hyperion runs rely on RDMA both for the latency-bound allreduces in the solver's inner loop and for the bulk transfers of halo regions between subdomains. RDMA-capable, GPU-aware transfers can move data from one node's GPU memory to another's without staging through host memory, which matters when the physics kernels live on the GPU.