Overlapping Communication and Computation
Overlapping hides communication latency by performing independent computation while messages are in flight rather than waiting for them.
Hiding latency
Communication takes time during which the processor would otherwise sit idle. Overlapping starts a transfer, then does useful, independent work while the network moves the data, so the communication time is hidden behind computation rather than added to it. When the overlap is complete, the effective cost of the communication approaches zero. This is one of the most reliable ways to improve strong scaling.
How it is done
Overlap requires nonblocking operations that return immediately and complete asynchronously: nonblocking MPI sends and receives (Isend/Irecv), nonblocking collectives (Iallreduce), and GPU streams for asynchronous transfers. The pattern is: post the communication, compute on the part of the problem that does not depend on the incoming data, then wait for the communication to finish and compute on the part that did depend on it.
- Post nonblocking communication, then do independent work, then wait.
- The classic split: update interior cells while halos are in transit.
- Requires genuinely independent work to fill the communication time.
- Needs a runtime or hardware that progresses transfers asynchronously.
The interior/boundary split
The canonical example is a domain-decomposed stencil. Each subdomain needs neighbor data (halo) to update its boundary cells but not its interior cells. So the code posts the halo exchange, updates the interior (which needs no remote data) while the halo travels, then updates the boundary once the halo arrives. If the interior work takes at least as long as the exchange, the communication is fully hidden.
A caveat
Overlap only helps if the runtime actually advances the transfer in the background; some MPI implementations make progress only inside MPI calls, so a long compute region may not overlap without a progress thread or periodic polling. And there must be enough independent work to hide behind. For a Hyperion halo exchange, verifying real overlap (not just posting nonblocking calls) is part of confirming a scaling improvement.