HPC & Compute

CUDA and GPU Programming

CUDA is the programming model for NVIDIA GPUs; it exposes threads, blocks, and a memory hierarchy that the programmer maps their computation onto.

A model for many threads

CUDA lets a programmer write a kernel, a function that runs on the GPU, and launch it across a grid of thread blocks. Each thread computes its global index from its block and thread coordinates and typically handles one data element. The runtime maps blocks onto the GPU's streaming multiprocessors.

Kernel shape

Kronos motion — pid vs model

python

# CUDA C pseudocode for c = a + b
# __global__ void add(float* a, float* b, float* c, int n){
#   int i = blockIdx.x * blockDim.x + threadIdx.x;
#   if (i < n) c[i] = a[i] + b[i];
# }
# launch: add<<<(n+255)/256, 256>>>(a, b, c, n);

The memory hierarchy

Registers: per-thread, fastest, scarce
Shared memory: per-block, on-chip, for cooperation and reuse
Global memory: large, high-bandwidth, high-latency device DRAM
Host memory: reached only over the PCIe or NVLink bus

Performance rules of thumb

Coalesce global-memory accesses so a warp reads contiguous addresses in one transaction. Reuse data through shared memory to cut trips to global memory. Keep enough warps resident to hide latency (occupancy), and avoid divergent branches within a warp. Minimize host-device transfers, which are slow relative to on-device bandwidth.

The wider ecosystem

Beyond hand-written kernels, libraries such as cuBLAS and cuFFT, and portable models like OpenMP offload, SYCL, and HIP, target GPUs. Deep-learning frameworks generate CUDA under the hood. The same principles, maximize arithmetic per byte moved and keep the device busy, apply regardless of the layer used.