In Step 3, each token produced one query (Q), one key (K), and one value (V) — all 16-dimensional. The model had one way to decide what to attend to. If it learned to focus on the immediately preceding character, that was its only strategy.

But language has multiple kinds of structure simultaneously. In the name “emma”, predicting what follows “a” might benefit from knowing:

• the character just before
• the first character
• the overall position

One head can’t attend to all three at once — it produces a single set of weights. Multi-head attention solves this by running multiple attention heads in parallel, each looking at a different slice of the embedding.

These patterns aren’t programmed — they’re learned during training. We don’t tell any head what to look for. The model discovers which attention strategies are useful for predicting the next character, and different heads end up specialising in different patterns.

The split

Instead of one 16-dimensional attention, we run four 4-dimensional attentions:

The Q, K, V projections still produce 16-dimensional vectors (same weight matrices). But instead of computing attention over all 16 dimensions at once, we partition them into four groups of 4:

• Head 0 uses dimensions 0–3
• Head 1 uses dimensions 4–7
• Head 2 uses dimensions 8–11
• Head 3 uses dimensions 12–15

Each head computes its own attention scores and weighted sum independently. Then the four 4-dimensional outputs are concatenated back into a 16-dimensional vector.

The committee analogy

Think of it as a committee of four specialists reviewing the same document. Each specialist reads the same text but focuses on different aspects — one might track syntax, another semantics, another position. They each write a 4-dimensional summary. The final report concatenates all four summaries into a 16-dimensional picture that’s richer than any single reviewer could produce.

The total computation is roughly the same as single-head attention (four heads × 4 dimensions ≈ one head × 16 dimensions), but the model can learn different attention patterns in each head. This is strictly more expressive — the model could learn to make all heads identical (recovering single-head behaviour), but it can also learn to specialise them.