Fine-Tuning Pretrained Encoders

Fine-tuning adapts a pretrained encoder to a specific task by attaching a small head and continuing training with a low learning rate on labeled data.

The transfer recipe

A pretrained encoder such as BERT already encodes broad linguistic structure. Fine-tuning specializes it. The standard procedure adds a task head on top of the encoder outputs, initializes it randomly, and trains the combined model on the target dataset. Because the encoder starts near a good solution, training is short, often a few epochs, and uses a small learning rate so the pretrained weights are nudged rather than overwritten.

Choosing what the head reads

• Sentence classification reads the pooled [CLS] representation
• Token tagging reads each token's final hidden state independently
• Span extraction predicts start and end positions over the sequence
• Sentence-pair tasks concatenate two inputs separated by [SEP]

Stability tactics

Fine-tuning small datasets can be unstable across random seeds. Common remedies include a brief learning-rate warmup followed by linear decay, gradient clipping, and layer-wise learning-rate decay that lets upper layers move more than the general-purpose lower layers. Freezing the lower layers entirely and training only the head is the cheapest option and a reasonable baseline when labeled data is scarce.

import torch
opt = torch.optim.AdamW(model.parameters(), lr=2e-5, weight_decay=0.01)
for epoch in range(3):
    for x, y in loader:
        loss = model(x, labels=y).loss
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        opt.step(); opt.zero_grad()

Parameter-efficient alternatives

Full fine-tuning updates every weight and produces a full copy of the model per task. Parameter-efficient methods instead train a small number of added parameters while freezing the backbone: low-rank adapters inject trainable low-rank matrices into the attention and MLP projections, and prompt tuning learns a few input vectors. These cut memory and let one frozen backbone serve many tasks.

The optimizer of choice is usually AdamW, and the schedule usually pairs warmup with decay as described in learning-rate schedules. The result is a task-specific model that inherits the general knowledge of the pretrained encoder while adding only what the task requires.