Distributed-Memory Parallelism
In distributed-memory systems each process owns private memory, and processes cooperate by exchanging explicit messages over a network.
Private memory, explicit messages
A distributed-memory program runs as many processes, each with its own private address space. No process can read another's memory directly; to share data they send messages. This model matches cluster hardware, where nodes are connected by a network rather than a shared memory bus, and it scales to the largest machines.
The dominant standard
MPI, the Message Passing Interface, is the near-universal library for distributed-memory programming. It provides point-to-point sends and receives plus collective operations such as broadcast and all-reduce that involve many processes at once.
Cost model
A message's time is roughly latency plus size divided by bandwidth. Latency is the fixed startup cost per message; bandwidth caps the transfer rate for large messages. Because latency dominates small transfers, efficient codes send fewer, larger messages and overlap communication with computation.
Decomposing the problem
- Partition the data across processes with domain decomposition
- Exchange only boundary (halo) data between neighbors
- Keep the partition balanced so no process lags
Strengths and demands
Distributed memory scales to millions of cores because there is no shared resource to contend for. The price is programmer effort: data placement, communication, and consistency are all explicit. Most large simulations, including plasma and neutron-transport codes used in fusion design, are built this way, usually as a hybrid with shared-memory threading inside each node. Because nothing is shared implicitly, distributed-memory bugs tend to be communication mistakes, a mismatched send and receive or a forgotten boundary exchange, rather than the timing races that plague shared memory.