Model Pruning
Pruning removes redundant weights or structures from a trained network to make it smaller and faster.
Cutting away redundancy
Neural networks are typically over-parameterized: many weights contribute little to the output. Pruning removes these redundant parameters, or whole structures such as channels and layers, to produce a smaller, faster model. The observation that a large fraction of weights can be deleted with minimal accuracy loss motivates pruning as a core model-compression technique.
Unstructured versus structured
Unstructured pruning zeroes out individual weights, usually those with the smallest magnitude, producing a sparse weight matrix. It can remove a large fraction of parameters but needs specialized hardware or libraries to turn sparsity into real speedup. Structured pruning removes entire channels, filters, attention heads, or layers, yielding a genuinely smaller dense model that runs faster on ordinary hardware, at the cost of coarser granularity.
The prune-and-recover cycle
Aggressive one-shot pruning hurts accuracy, so pruning is typically iterative: prune a portion, fine-tune to recover accuracy, and repeat, gradually reaching high sparsity. Magnitude is the most common importance criterion, but methods also use gradient or sensitivity information to decide what to remove.
- Magnitude pruning: remove the smallest-magnitude weights
- Iterative pruning with fine-tuning preserves accuracy better than one-shot
- Structured pruning gives portable speedups; unstructured gives higher compression
The lottery ticket hypothesis
A notable finding, the lottery ticket hypothesis, holds that a dense network contains a small sub-network which, trained from the original initialization, can match the full model accuracy. This suggests the value of a large network lies partly in providing many candidate sub-networks to find. Pruning combines naturally with quantization and distillation in a compression pipeline that fits capable models onto constrained devices.
As with all compression, the pruned model must be re-evaluated on held-out data, since sparsity can affect robustness and calibration, not just top-line accuracy.