HPC & Compute

MPI: The Message Passing Interface

MPI is the standard library for distributed-memory parallel programming, defining how processes exchange data through messages and collectives.

What MPI is

MPI is a specification, implemented by libraries such as Open MPI and MPICH, for writing programs that run as many cooperating processes. Each process has a unique rank within a communicator (a group of processes), and knows the group's size. Ranks decide which part of the work they own and communicate by calling MPI functions. The same executable runs on every rank, and code branches on the rank number to divide the work, a style known as single-program, multiple-data.

The core operations

Kronos motion — data assimilation

MPI_Send / MPI_Recv: point-to-point transfer between two ranks
MPI_Bcast, MPI_Scatter, MPI_Gather: one-to-many and many-to-one
MPI_Reduce / MPI_Allreduce: combine values (sum, max) across ranks

A minimal program

python

from mpi4py import MPI
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
size = comm.Get_size()
# each rank contributes its rank number; all get the sum
total = comm.allreduce(rank, op=MPI.SUM)
if rank == 0:
    print('sum of ranks =', total)  # 0+1+...+(size-1)

Blocking versus non-blocking

Blocking calls return only when it is safe to reuse the buffer. Non-blocking calls (MPI_Isend, MPI_Irecv) return immediately and are completed later with MPI_Wait, letting a program overlap communication with computation. This overlap is often the difference between a code that scales and one that stalls.

Why it endures

MPI is portable across nearly every HPC system and maps cleanly onto fast interconnects. It is verbose and low-level, but it gives explicit control over data movement, which is what large-scale performance demands. Most production simulation codes, in fusion and elsewhere, are built on MPI, frequently combined with OpenMP or GPU code inside each rank.