Pointer Networks

Pointer networks use attention as the output itself, selecting positions in the input sequence rather than tokens from a fixed vocabulary.

Outputs that point

Ordinary sequence-to-sequence models produce outputs from a fixed vocabulary decided before training. Some problems have outputs that are positions in the input, and whose valid range depends on the input length, so a fixed vocabulary does not fit. Pointer networks solve this by repurposing the attention mechanism: instead of using attention weights to blend a context vector, the network uses the attention distribution over input positions directly as its output, pointing at whichever input element to select next.

Attention as a selector

At each output step the decoder computes an attention score for every input position, comparing its current state against each input's encoding. A softmax over these scores yields a distribution over input positions, and the highest, or a sampled one, is the chosen output. Because the number of positions equals the input length, the output space automatically grows and shrinks with the input, which a standard classifier over a fixed vocabulary cannot do.

# at decoder step, score each input position i
u_i = v.tanh(W1 @ enc_i + W2 @ dec_state)   # scalar per input position
p = softmax(u)                               # distribution over positions
choice = p.argmax()                          # point to an input element

• Output vocabulary is the input positions, so it scales with input size
• No fixed maximum output symbol set
• Naturally suited to selecting, ordering, or subsetting the input
• Built entirely on the attention mechanism

Where they apply

Pointer networks fit combinatorial problems whose answer is an arrangement or selection of the inputs, such as finding a convex hull, ordering points for a traveling-salesman-style tour, or extractive summarization where the output words are copied from the source. The same copy mechanism was later folded into general sequence models as a copy or pointer-generator hybrid, letting a model either generate a vocabulary token or point to and copy an input token. The idea grew from the attention used in memory-augmented and translation models.