Foundation Models
Foundation models are large networks pretrained on broad data that adapt to many downstream tasks, a shift from task-specific to general-purpose AI.
A change in paradigm
A foundation model is a single large model trained on broad, mostly unlabeled data that can be adapted to a wide range of tasks. The term, coined in 2021, marks a shift from building a separate model per task to building one general model and specializing it. Most foundation models are transformers, trained with self-supervised objectives such as next-token or masked-token prediction that need no manual labels, so they can consume enormous datasets.
Self-supervised pretraining
The key enabler is self-supervision: the training signal comes from the data itself. Predicting the next word, filling in a masked patch, or matching an image to its caption creates supervision at scale without human annotation. This lets the model learn broad structure from raw text, images, or other signals, building representations that serve as a general foundation for later adaptation.
Adaptation methods
- Fine-tuning: continue training the model's weights on a specific task.
- Parameter-efficient tuning: update a small number of added parameters (adapters, LoRA) while freezing the base.
- Prompting: describe the task in the input, with no weight changes.
- In-context learning: give examples in the prompt so the model infers the task on the fly.
Emergent capabilities and scaling
As foundation models grow in parameters, data, and compute, performance improves along smooth scaling laws, and some abilities appear only past a size threshold, such as multi-step reasoning or following novel instructions. This predictable improvement with scale, combined with occasional qualitative jumps, drove the rapid growth of model size and the concentration of capability in a few large base models.
Reach, limits, and use in science
Foundation models now span language, vision, code, audio, and multimodal combinations, and specialized ones model proteins, chemistry, and climate. Limitations are real: they can produce confident but false statements, inherit biases from training data, are costly to train, and can be opaque. In research and engineering they act as flexible assistants and as backbones fine-tuned on domain data, for example a model adapted to a body of technical literature or simulation output to support analysis, always with human verification of the results.
- One broadly pretrained model adapts to many tasks.
- Self-supervision provides labels for free at scale.
- Adapted by fine-tuning, lightweight tuning, or prompting.
- Powerful but prone to errors, bias, and high cost.