Self-Supervised Learning
Self-supervised learning creates its own labels from unlabeled data by hiding part of the input and predicting it.
Supervision from the data itself
Self-supervised learning trains models on unlabeled data by inventing a prediction task from the data structure, no human annotation required. The idea is to withhold part of each example and train the model to recover it, forcing it to learn representations that capture how the data is organized. Those representations then transfer to downstream tasks with far fewer labels.
Pretext tasks
- Masked prediction: hide tokens or patches and predict them, the basis of masked language and masked image modeling
- Next-element prediction: predict the next word or frame, the autoregressive objective behind large language models
- Context prediction: predict the relative position of image patches or the ordering of shuffled inputs
- Denoising: corrupt the input and reconstruct the clean version
Why it works
To predict a masked word well, a model must learn syntax, semantics, and world knowledge; to fill in a masked image region, it must learn shapes, textures, and object structure. The pretext task is a means, not the end: after pretraining, the model is fine-tuned or probed on the real task, where the learned features provide a strong starting point.
Impact and relation to contrastive learning
Self-supervision is the engine behind modern foundation models: pretraining on vast unlabeled corpora, then adapting to specific tasks. It dramatically reduces the labeled data needed downstream. Contrastive learning is a major branch that builds its pretext task from similarity between augmented views rather than reconstruction. Together they made large-scale representation learning practical where labels are scarce.
Evaluation typically uses linear probing (freeze the representation, train a linear classifier) to isolate representation quality from downstream capacity, alongside full fine-tuning.