Each token looks back at the others.

A token’s embedding alone can’t tell “bank” the river from “bank” the money. Attention lets each token mix in information from other tokens, weighted by how relevant they are, so “bank” ends up carrying a bit of “river”.

Each token makes three lists from its embedding: a query (what it’s looking for), a key (what it offers to be found by) and a value (what it passes on). “bank”’s query is compared with every key by a dot product; the scores are divided by √d (here d = 4) and turned into shares by softmax; the shares mix the values. The numbers below are small and made up so you can follow them; a real model uses hundreds per head.

Softmax raises e to each score and divides by the total, so big scores get most of the share and every share stays positive. Dividing by √d first keeps scores from growing with the size of the vectors; without it, softmax would give nearly everything to one token and training would stall.

Models that write (decoders, like GPT and Qwen) use a causal mask: a token attends to itself and earlier tokens only. Models that read a whole text at once (encoders, like BERT) attend in both directions, which suits classifying or embedding text but not writing it word by word.

A head is one query-key-value set. Several heads in a layer can each follow something different (researchers have found heads that track the previous token, or a verb’s object), and stacking layers lets later ones build on earlier ones. Qwen 3 (8B) shares each key and value between groups of query heads: 32 query heads but 8 key-value heads, which the next lessons come back to.

The scores grow with the square of the length. At 32,768 tokens that’s about 537 million scores per head per layer. Much of the engineering in modern models, from the KV cache to faster attention kernels, is about this cost.