Network Topologies
Topology is the arrangement of links between nodes; it determines message hop counts, bisection bandwidth, and how a cluster scales.
Why arrangement matters
Given many nodes and a budget of links and switches, the topology decides how far apart nodes are and how much bandwidth is available when many pairs communicate at once. Two figures of merit dominate: the diameter (worst-case hop count between nodes) and the bisection bandwidth (aggregate bandwidth across a cut that splits the machine in half).
Common designs
- Fat-tree: a layered tree with more bandwidth toward the root; near full bisection bandwidth
- Torus (2D/3D): nodes in a mesh with wrap-around links; short diameter for neighbor communication
- Dragonfly: groups of densely connected nodes with sparse global links; low diameter at large scale
The trade-off
Full bandwidth between every pair (a non-blocking fat-tree) is expensive, and cost grows faster than node count. Torus and dragonfly designs accept lower worst-case bandwidth for far fewer cables, betting that most communication is either local (neighbor exchanges) or tolerant of some contention.
Matching code to topology
A stencil code that only exchanges with neighbors thrives on a torus, since its communication is inherently local. A code with heavy all-to-all or global collectives stresses bisection bandwidth and prefers a fat-tree. Rank placement, mapping MPI ranks onto physically close nodes, can improve either case.
Contention and routing
Even a good topology suffers if many messages route through the same links, causing contention. Adaptive routing spreads traffic across alternative paths, and job schedulers try to allocate contiguous, well-shaped node sets so that a job's interconnect traffic stays local and predictable.