Warp Divergence
When threads in a warp take different branches, the hardware runs each path serially, wasting throughput on the inactive threads.
Why divergence costs
All 32 threads in a warp share one program counter. When a data-dependent branch sends some threads down the if path and others down the else path, the hardware cannot run both at once. It executes the taken path with the other threads masked off, then executes the other path with the first group masked off. Both paths run in sequence, so a fully divergent branch can halve throughput; nested divergent branches compound the loss.
Where it appears
Divergence arises whenever a branch condition depends on thread-specific data: particles in different physical regions taking different code paths, boundary versus interior cells in a stencil, or early-exit conditions that trigger for only some threads. A loop whose trip count varies per thread also diverges, since the warp must keep iterating until the longest-running thread finishes.
- A warp serializes over each distinct branch path its threads take.
- Cost scales with how many paths and how deeply nested they are.
- Uniform branches (same for all 32 threads) are essentially free.
- Predication can replace short branches with masked execution.
Reducing it
Common remedies: sort or group data so threads in a warp are likely to take the same path (bucketing particles by region before the kernel); restructure algorithms so the branch is on a value uniform across the warp; and let the compiler use predication for short branches, where both sides compute and a mask selects the result, avoiding a real branch. Moving divergent decisions to the block or grid level, so an entire warp handles one case, also helps.
In practice
A Monte Carlo neutron transport kernel for a Hyperion run can suffer heavy divergence when neighboring threads track particles undergoing different interactions. Grouping particles by interaction type between kernel launches keeps each warp more uniform, recovering throughput that naive per-particle branching would lose.