The Attention Mechanism

Attention lets a model dynamically weight different parts of its input, focusing on what is relevant to each step of its output.

The core idea

Attention computes a weighted average of a set of values, where the weights depend on how relevant each value is to a given query. Instead of forcing a model to compress everything into one fixed representation, attention lets it retrieve the pieces it needs on demand. The mechanism generalizes soft lookup: a query is compared to a set of keys, and the resulting compatibility scores decide how much of each associated value to read.

Query, key, value

Kronos motion — what is fusion

The three roles borrow from database retrieval. The query is what the current step is looking for. Each key describes a candidate item, and each value is the content to retrieve if that item is chosen. A similarity between query and key sets the weight, and the output is the weighted sum of values. Unlike a hard lookup, attention returns a soft blend across all items.

Scaled dot-product attention

The dominant form scores a query against keys by dot product, divides by the square root of the key dimension to keep values in a stable range, applies softmax to turn scores into weights that sum to one, and multiplies by the values. Compactly: Attention(Q, K, V) = softmax(Q·K^T / sqrt(d)) · V. The scaling matters because large dot products would push softmax into saturated regions with tiny gradients.

python

import numpy as np
def attention(Q, K, V):
    d = K.shape[-1]
    scores = Q @ K.T / np.sqrt(d)
    scores -= scores.max(-1, keepdims=True)
    w = np.exp(scores); w /= w.sum(-1, keepdims=True)
    return w @ V

Why it changed everything

Attention first appeared as an add-on to recurrent seq2seq models, removing the fixed-vector bottleneck by letting the decoder look at every encoder state. Its success led to the realization that attention alone, without recurrence or convolution, could model sequences. That insight produced the transformer and, through it, modern large language models.

Interpretability caveat

Attention weights are sometimes read as explanations of what the model focused on. This is only loosely valid: high weight indicates one signal among many inside a deep network, and the same output can arise from different weight patterns. Attention maps are a useful diagnostic, not a definitive account of the model's reasoning.

A weighted average whose weights measure relevance.
Query-key-value framing generalizes soft lookup.
Scaled dot-product with softmax is the standard form.
Attention alone, without recurrence, powers transformers.