GPU Memory Hierarchy
GPUs expose several memory spaces, from fast per-block shared memory to large but distant global memory, each with distinct latency and bandwidth.
The levels
A GPU's memory is a hierarchy that the programmer partly controls. Registers are per-thread and fastest but scarce. Shared memory is a small, fast, software-managed scratchpad private to a thread block, used for data reuse and cooperation among threads. L1 and L2 caches sit between the SMs and device memory. Global memory (HBM or GDDR) is large, holds the bulk of the data, and has high bandwidth but high latency. Constant and texture memories are specialized read paths.
Shared memory as a managed cache
The distinguishing feature of GPU programming is explicit control of shared memory. A kernel can stage a tile of global data into shared memory once, then have all threads in the block reuse it many times at near-register speed. Matrix multiplication, stencils, and convolutions are all built around this tiling pattern. Shared memory is organized into banks; when threads hit different banks the accesses proceed in parallel, but two threads hitting the same bank cause a bank conflict that serializes.
- Registers: per-thread, fastest, limited count constrains occupancy.
- Shared memory: per-block scratchpad for reuse and cooperation.
- Global memory: large, high bandwidth, high latency, off-chip.
- Bank conflicts serialize shared-memory access and must be avoided.
Bandwidth is the budget
GPU kernels are far more often limited by memory bandwidth than by arithmetic throughput. Global memory bandwidth is high in absolute terms but modest relative to the enormous floating-point rate of the compute units. The ratio of the two sets the roofline. The optimization goal for most kernels is to move each byte from global memory as few times as possible and reuse it heavily from shared memory or registers.
Design consequence
Kernels are structured to maximize data reuse per byte fetched. A field-update stencil for a Hyperion mesh loads a tile plus its halo into shared memory, then every interior thread reuses neighboring values without returning to global memory, converting a bandwidth-bound access pattern into an on-chip one.