Profiling and Performance Analysis
Profiling measures where a program spends time and resources so optimization targets the true bottleneck instead of a guess.
Measure, do not guess
Performance intuition is unreliable; programs routinely spend most of their time somewhere other than where the author expects. Profiling instruments a run to record where time, memory traffic, and communication actually go, so effort is spent on the parts that matter. Optimizing an unmeasured code is guesswork.
Kinds of profiling
- Sampling: periodically record the call stack; low overhead, statistical
- Instrumentation: insert timers around regions; precise but intrusive
- Hardware counters: read CPU/GPU events (cache misses, FLOPs, stalls)
- Trace analysis: timeline of communication and synchronization across ranks
What to look for
A profile reveals hot functions, but the deeper questions are why they are slow. Is a kernel compute-bound or memory-bound? Are cores stalled on cache misses? Is one MPI rank waiting on a slow neighbor (load imbalance)? Are collectives dominating? Each answer points to a different fix.
Tools
General profilers (perf, gprof, VTune), MPI trace tools (Score-P, TAU, Vampir), and vendor GPU profilers (Nsight, rocprof) cover the layers. Roofline analysis tools place a kernel against the hardware's arithmetic and bandwidth ceilings, showing how much headroom remains and whether the limit is compute or memory.
The optimization loop
Effective tuning is iterative: profile to find the bottleneck, apply the targeted change, re-profile to confirm it helped and to expose the next limit. Because fixing one bottleneck reveals another, a single measurement is never enough. Stop when further effort no longer moves time to solution meaningfully.