Self-Attention

Self-attention lets every position in a sequence attend to every other, building context-aware representations in a single parallel operation.

Attention within a sequence

In self-attention, the queries, keys, and values all come from the same sequence. Each token generates a query and compares it to the keys of every token, itself included, to decide how much to read from each token's value. The output for each position is a context-aware representation that blends information from the whole sequence, weighted by relevance.

The computation

Kronos motion — parallel

From input X the model computes Q = X·Wq, K = X·Wk, V = X·Wv using learned projection matrices. It then applies scaled dot-product attention: softmax(Q·K^T / sqrt(d))·V. The result is a new sequence of the same length where each vector has been enriched by the tokens it attended to. Because it is matrix multiplication, all positions are processed at once.

Parallelism and range

Unlike a recurrent network, self-attention has no sequential dependency across positions, so it fully parallelizes on modern hardware. It also connects any two positions in a single step, giving a constant path length between distant tokens rather than the linear path an RNN imposes. This is why transformers capture long-range dependencies more effectively and train faster than recurrent models.

The quadratic cost

Every position attends to every other, so the attention matrix has size proportional to the square of the sequence length. Memory and computation grow quadratically, which limits how long a sequence can be processed directly. A large research effort produces efficient variants, such as sparse, low-rank, and windowed attention, that approximate full self-attention at reduced cost for long inputs.

python

import numpy as np
def self_attention(X, Wq, Wk, Wv):
    Q, K, V = X @ Wq, X @ Wk, X @ Wv
    s = Q @ K.T / np.sqrt(K.shape[-1])
    s -= s.max(-1, keepdims=True)
    w = np.exp(s); w /= w.sum(-1, keepdims=True)
    return w @ V

Masking

In autoregressive models a causal mask blocks each position from attending to future positions, so predictions depend only on past tokens, which is essential for text generation. Encoders use no such mask and let every position see the full sequence. Padding masks prevent attention to filler tokens added to make batched sequences equal length.

Queries, keys, and values all come from one sequence.
Each token is re-represented using the whole context.
Fully parallel, with constant path length between positions.
Cost grows quadratically with sequence length.