WaveNet

WaveNet generates raw audio one sample at a time using a deep stack of dilated causal convolutions with gated activations, modeling waveforms directly.

Modeling raw audio

WaveNet is an autoregressive generative model of raw audio waveforms. Instead of predicting spectral features and reconstructing sound, it predicts the next audio sample directly from all previous samples. Audio is sampled tens of thousands of times per second, so a model must capture structure over very long ranges. WaveNet does this with a deep stack of dilated causal convolutions whose receptive field spans thousands of samples.

Gated activation units

Each layer uses a gated activation borrowed from the LSTM family: the convolution output is split into two paths, one passed through a tanh (the content) and one through a sigmoid (the gate), and the two are multiplied elementwise. The gate learns how much of each filter's response to let through, which improved audio quality over a plain rectified nonlinearity in the original work.

z = torch.tanh(conv_f(x)) * torch.sigmoid(conv_g(x))   # gated unit
skip = skip_proj(z)
out = x + res_proj(z)                                    # residual

Residual and skip connections

Every layer feeds a residual path that carries the signal to the next layer and a skip path that is summed across all layers to form the output representation. The skip connections let the final prediction draw on features from every depth, and the residual paths keep the deep stack trainable. The output is a categorical distribution over quantized sample values, trained with cross-entropy.

• Direct waveform modeling avoids hand-designed audio features
• Dilation gives a long receptive field without pooling
• Naive sample-by-sample generation is slow, since each sample needs a full pass
• Later variants distilled WaveNet into parallel generators for real-time synthesis

Legacy

WaveNet set a quality benchmark for text-to-speech and demonstrated that convolutional autoregressive models could rival and surpass recurrent approaches for long sequences. Its combination of dilated causal convolution, gating, and residual and skip connections influenced later sequence models and directly shaped temporal convolutional networks. The autoregressive framing it shares with language models is described in autoregressive modeling.