GPU Memory and Data Movement
A GPU's performance is usually set by its memory system, so managing device memory and host-device transfers is the key to speed.
The memory-bound reality
Most GPU kernels are limited not by arithmetic but by how fast data reaches the arithmetic units. A GPU has enormous compute throughput, so unless a kernel reuses data heavily, it runs at the pace of memory. Understanding the GPU memory hierarchy is therefore the first step in getting performance.
The hierarchy on a GPU
- Registers: per-thread, fastest, limited; over-use spills to slow memory
- Shared memory / L1: per-block, on-chip, for cooperation and data reuse
- L2 cache: shared across the device
- Global memory (HBM): large and high-bandwidth but high-latency
- Host memory: reached only across the PCIe or NVLink bus
Coalescing
When the threads of a warp access consecutive global-memory addresses, the hardware combines them into a few wide transactions, coalesced access. Strided or scattered access breaks this into many small transactions, wasting bandwidth. Data layout that makes warp accesses contiguous is one of the most important GPU optimizations.
Host-device transfers
Moving data between CPU and GPU memory is slow relative to on-device bandwidth, and a naive code that shuttles data back and forth every kernel spends most of its time waiting. The remedy is to keep data resident on the device across many kernels and to overlap unavoidable transfers with computation using asynchronous copies and streams.
Reuse through shared memory
Shared memory lets a block stage a tile of data on-chip and reuse it across many threads, cutting global-memory traffic. This is the GPU analogue of cache blocking, and it is what turns a memory-bound kernel such as matrix multiply into a compute-bound one that approaches peak.