Dilated Causal Convolutions
Dilated causal convolutions insert gaps between kernel taps to widen the receptive field exponentially while keeping outputs strictly dependent on the past.
Two constraints, one operator
Sequence models built on convolution often need two properties at once. Causality means the output at time t must not depend on any future input, which is essential for autoregressive generation and honest forecasting. A wide receptive field means the model can reach far back in time. Dilated causal convolution satisfies both: the convolution is shifted so the kernel only covers past positions, and the kernel taps are spaced apart by a dilation factor.
How dilation works
A standard convolution with kernel size k reads k consecutive inputs. A dilated convolution with dilation d reads k inputs spaced d apart, so it covers a span of (k-1)*d + 1 while still using only k weights. Increasing d skips over positions, sampling a wider window at the same cost. Because the operator only looks backward, causality is preserved by padding on the left and cropping the right.
Stacking for exponential coverage
The power comes from stacking layers with geometrically increasing dilation, such as 1, 2, 4, 8. Each layer roughly doubles the reach of the one below, so the total receptive field grows exponentially with depth while the parameter count grows only linearly. A handful of layers can therefore cover thousands of time steps.
- Constant number of weights per layer regardless of dilation
- No pooling, so temporal resolution is preserved throughout
- Left padding keeps outputs aligned with inputs and strictly causal
- Exponential receptive-field growth from linear depth
Where they are used
Dilated causal convolutions are the backbone of temporal convolutional networks and of WaveNet, which generates raw audio sample by sample. The same idea appears wherever a model must respect time order yet reach far back cheaply, making it a general tool for forecasting, audio, and other ordered signals such as sensor and instrument time series.