Vision Transformer

The Vision Transformer applies the pure transformer to images by cutting them into patches and treating each patch as a token.

Images as sequences of patches

The Vision Transformer (ViT) shows that the transformer, designed for text, works on images with almost no change. An image is split into a grid of fixed-size patches, for example 16x16 pixels. Each patch is flattened and linearly projected into a token vector. The resulting sequence of patch tokens is fed to a standard transformer encoder, exactly as a sentence of word tokens would be.

The classification token

Kronos motion — transformer

Following BERT, ViT prepends a special learnable token, often called the class token. After the transformer processes the sequence, this token's output vector aggregates information from all patches and is fed to a small head for the final prediction. Positional embeddings are added to patch tokens so the model knows each patch's location in the grid, since attention is otherwise order-blind.

python

# ViT input pipeline (sketch)
# image (H,W,3) -> patches (N, P*P*3) -> tokens (N, d)
# tokens = [CLS] + patch_proj(patches) + position_embeddings
# output = transformer_encoder(tokens); logits = head(output[0])

Data appetite

Convolutional networks bake in strong priors: locality and translation equivariance. ViT has weaker built-in priors and must learn spatial structure from data, so it needs large training sets to match or beat CNNs. Pretrained on very large image collections, ViT surpasses comparable CNNs; trained from scratch on small datasets, it typically underperforms them. Techniques like strong augmentation and distillation reduce this data requirement.

Strengths

Because self-attention connects any two patches directly, ViT captures long-range spatial relationships more readily than a CNN, whose receptive field grows only with depth. It also scales smoothly with model and data size, inheriting the transformer's favorable scaling behavior. This makes ViT and its variants the backbone of many state-of-the-art vision and multimodal systems.

Variants and hybrids

Hierarchical designs such as the Swin Transformer reintroduce locality with windowed attention and produce multi-scale feature maps suited to detection and segmentation. Hybrid models use a few convolutional layers to produce tokens before the transformer. For grid-structured scientific imagery, such as diagnostic camera frames or 2D field maps, patch-based transformers offer a way to relate distant regions that pure convolution reaches only through many layers.

Images become sequences of patch tokens.
A class token aggregates the final representation.
Needs large data due to weaker built-in priors.
Captures long-range spatial relations directly.