Transfer Learning

Transfer learning reuses knowledge from a model trained on a large dataset to solve a related task with far less data.

The core idea

Training a deep network from scratch needs large amounts of data and computation. Transfer learning avoids that by starting from a model already trained on a big, general dataset and adapting it to a new, usually smaller, task. The pretrained model has learned broadly useful features, edges and textures for images, grammar and semantics for text, and these transfer to related problems, so the new task needs only modest data and training.

Feature extraction versus fine-tuning

Kronos motion — pid vs model

Feature extraction: freeze the pretrained network and train only a new output head on top. Fast, low-data, treats the network as a fixed feature computer.
Fine-tuning: continue training some or all pretrained weights on the new task, usually with a small learning rate. Higher accuracy but needs more data and care to avoid overwriting useful features.

Why features transfer

Deep networks learn a hierarchy: early layers capture general, task-agnostic patterns, while later layers capture features specific to the original task. General early features are broadly reusable, so a common strategy is to keep early layers frozen and adapt later ones. The more similar the new task is to the original, the more layers can be reused unchanged.

python

import torchvision.models as m
net = m.resnet50(weights='IMAGENET1K_V2')
for p in net.parameters():
    p.requires_grad = False        # freeze backbone
net.fc = torch.nn.Linear(2048, num_classes)  # new trainable head

The pretrain-then-adapt paradigm

Transfer learning is now the default across the field. In vision, models pretrained on large image collections are adapted to specific classification, detection, and segmentation tasks. In language, models pretrained on large text corpora are fine-tuned or prompted for downstream tasks. The expensive pretraining is done once, and its cost is amortized over many cheap adaptations.

Practical benefits and cautions

Transfer learning improves accuracy on small datasets, cuts training time and compute, and often produces more robust models. Cautions include domain shift, where the source and target data differ enough that transferred features are misleading, and catastrophic forgetting, where aggressive fine-tuning erases useful pretrained knowledge. In scientific settings, a model pretrained on abundant simulation data can be fine-tuned on scarce experimental measurements, provided the two domains are close enough for the features to carry over.

Reuse a pretrained model to cut data and compute needs.
Feature extraction freezes; fine-tuning adapts weights.
General early features transfer better than specific late ones.
Watch for domain shift and catastrophic forgetting.