Neural networks are, at their core, big piles of matrix multiplication. They consume numbers and emit numbers. So before anything interesting can happen, we need to answer a deceptively simple question: how do you feed text into a pile of math?

Step 1: Tokenization — text → integers

A token is a chunk of text — often a word, sometimes a piece of a word like trans + former, sometimes punctuation. The model has a fixed dictionary of every token it knows, called the vocabulary (typically 30,000–100,000 entries). A tokenizer is just a deterministic function that splits text and looks up each chunk's ID:

// conceptually:
tokenize("the cat sat on the mat")
// → ["the", "cat", "sat", "on", "the", "mat"]
// → [101, 5432, 8871, 209, 101, 6310]   (vocabulary IDs)

Step 2: Embeddings — integers → vectors

An integer ID carries no meaning. Token 5432 isn't "more" than token 209. So the model immediately swaps each ID for an embedding: a learned vector of floating-point numbers, typically 512 to 12,000+ dimensions long.

// the embedding table: one row per vocabulary entry
float embeddings[VOCAB_SIZE][D_MODEL];   // e.g. [50000][512]

// "embedding lookup" is literally array indexing:
vector_for_cat = embeddings[5432];        // 512 floats

Here's the magic part: those floats are learned during training, and they end up encoding meaning. Words used in similar contexts drift toward similar vectors. "cat" and "dog" end up close together; "cat" and "carburetor" end up far apart.

Measuring "similar": the dot product

If meaning lives in vectors, we need a way to compare vectors. The workhorse is the dot product: multiply matching elements, then sum.

Two vectors pointing in similar directions produce a large positive dot product. Unrelated directions produce a number near zero. Opposite directions go negative. Hold onto this — the entire attention mechanism is built on "dot product = relevance score."

One problem we haven't solved

The embedding for "cat" is the same vector every time, in every sentence. But meaning depends on context: "bank" in "river bank" vs "bank account" deserves different representations. And word order matters: "dog bites man" ≠ "man bites dog". Static, per-word vectors can't capture either. That problem is what the rest of this course solves.

You don't need deep RNN knowledge to read the Transformer paper — but you do need to know what the paper was reacting against. The first sentence of the abstract calls out "complex recurrent or convolutional neural networks." Here's the 10-minute version of that backstory.

Before 2017: process tokens one at a time

A Recurrent Neural Network (RNN) handles a sequence the way you'd process a stream: one item per iteration, carrying state forward.

let hidden = initialState;          // ONE fixed-size vector, e.g. 512 floats
for (token of sentence) {
  hidden = cell(hidden, token);      // fold the new token into the state
}
// `hidden` is now the "summary" of the whole sentence

It's a fold/reduce over the sequence. Elegant — and it powered machine translation for years (LSTMs and GRUs are fancier versions of cell() with learned "what to keep / what to forget" gates). But the design has two structural flaws.

Flaw 1: the memory bottleneck

Everything the model knows about tokens 1…N must be crammed into that single fixed-size hidden vector. By token 40, the contribution of token 1 has been overwritten and diluted dozens of times. Long-range connections — like a pronoun referring to a noun 30 words back — get lossy. (Training-wise this shows up as the vanishing gradient problem: the learning signal fades as it's back-propagated through many steps.)

Flaw 2: it's inherently serial

Step t needs the output of step t−1. You cannot parallelize the loop. GPUs are throughput monsters that want to do thousands of operations at once, and the RNN forces them to walk single-file. Training on web-scale data was painfully slow.

The 2017 proposal

The paper's title is the thesis: throw away recurrence entirely. Replace it with a mechanism — attention — where each token can directly look at every other token, no matter the distance, and where all tokens are processed simultaneously. One token away or 500 tokens away: same single hop. And because nothing depends on the previous loop iteration, the whole thing is giant matrix multiplications, which GPUs devour.

Forget the math for one module. Attention answers a single question, asked once per token:

Take the sentence the paper itself made famous:

“The animal didn't cross the street because it was too tired.”

What does it refer to? You instantly know: the animal (streets don't get tired). A model needs the same skill. With attention, when the model processes it, it computes a relevance score against every word in the sentence, and ideally puts most of its weight on animal.

Attention output = a weighted average

Concretely, attention does this for each token:

1. Score the current token against every token in the sentence (one number each).
2. Normalize the scores so they're positive and sum to exactly 1 — now they're percentages. This normalization is called softmax, and it also exaggerates the gaps: clear winners get most of the mass.
3. Build the token's new representation as the weighted average of all tokens' vectors, using those percentages.

So after attention, the vector for it is no longer the generic dictionary entry for "it" — it's mostly a blend of animal, with traces of tired and others mixed in. The token has absorbed its context. This is why attention output is called a contextual representation, and it's exactly the fix for Module 1's "static embedding" problem.

Why "self"-attention?

When a sentence attends to itself — every token scoring every token of the same sequence — it's called self-attention. Later (Module 7) you'll meet cross-attention, where one sequence (a translation being written) attends to a different one (the source sentence). Same machinery, different participants.

What's missing from this picture

We hand-waved "score the current token against every token." Scored how? You might guess: dot product of their embeddings. Close — but the real design is cleverer, and it's the heart of the paper: each token plays three different roles via three different vectors. That's Q, K, and V. Next module.

This is the module the whole course bends toward. Once Q·K·V clicks, the rest of the paper is plumbing.

Three roles for every token

In Module 3, each token both asked for context and provided context. Those are different jobs, so the Transformer gives each token three separate vectors, produced from its embedding x by three learned weight matrices:

q = x · WQ   // Query: "what am I looking for?"
k = x · WK   // Key:   "what do I advertise / how can I be found?"
v = x · WV   // Value: "what content do I hand over if matched?"

The matrices W_Q, W_K, W_V are the learned parameters — the same three matrices applied to every token. Training tunes them so that useful queries find useful keys.

The recipe, end to end

For the token it in our running sentence:

1. Score: dot its query against every token's key: score_j = q_it · k_j. High score = "this key matches what I'm looking for."
2. Normalize: softmax the scores into weights that sum to 1.
3. Blend: output = Σ weight_j · v_j — a weighted sum of value vectors.

Every token does this simultaneously. Stack all the q's into a matrix Q (one row per token), likewise K and V, and steps 1–3 collapse into two matrix multiplications and a softmax. That's why this is fast on GPUs.

Now you can read the paper's main equation

• Q Kᵀ — every query dotted with every key, all at once. For n tokens this yields an n × n score matrix: cell (i, j) = how much token i cares about token j.
• softmax(...) — applied per row, turning each row of scores into weights summing to 1.
• ... V — multiplying by V computes every token's weighted blend of values in one shot.
• √dk — a scaling factor we'll justify in the next module.

Why divide by √d_k?

Recall the dot product sums one term per dimension. With d_k = 64 dimensions, you're summing 64 terms; with more dimensions, raw scores naturally grow larger just because there are more terms — not because matches are more meaningful. Feed huge scores into softmax and it saturates: one weight pins to ≈1.0, everything else to ≈0.0.

A saturated softmax is nearly flat, so its gradients are nearly zero — and gradients are the only learning signal the network gets. Dividing scores by √d_k keeps them in a healthy range regardless of dimension. (Why square root? If vector entries are roughly unit-variance, the dot product's standard deviation grows as √d_k, so this division renormalizes it back to ~1.)

One head can only look one way

A single attention pass produces one weighting per token — one "way of relating." But language is multi-relational. In our sentence, it should connect to animal (what it refers to), but also to tired (what's predicated of it), while other relations (syntax, adjacency, punctuation structure) matter for other tokens. A single weighted average has to mush all of those together.

The fix: run h heads in parallel

Instead of one attention with full-width vectors, the paper runs h = 8 heads, each with its own smaller W_Q, W_K, W_V (each head works in 512/8 = 64 dimensions). Each head learns its own notion of relevance. Then the 8 outputs are concatenated back to 512 dims and passed through one more learned matrix, W_O, to mix them.

When researchers inspect trained models, they find heads that really do specialize: heads tracking subject–verb pairs, heads that always attend to the previous token, heads resolving pronouns. Nobody programs this — it emerges from training.

Attention has a hidden flaw, and it's a direct consequence of its biggest strength. Because every token attends to every other token symmetrically — no loop, no left-to-right walk — the mechanism is order-blind. Shuffle the input tokens and QKᵀ produces the same scores, just shuffled. To pure attention, a sentence is a set, not a sequence. "dog bites man" and "man bites dog" would be indistinguishable.

RNNs got order for free (the loop was the order). Having deleted the loop, the Transformer must inject position back in as data.

The solution: add a position signature to each embedding

Before the first attention layer, the model adds a positional encoding — a vector that depends only on the position index — to each token's embedding:

input[i] = embedding(token[i]) + positional_encoding(i)

Now the vector for "dog" at position 0 differs slightly from "dog" at position 3, and attention heads can learn to use that difference (e.g. "attend to the previous token" — Head 2 from last module — is only possible because position is in the vectors).

The sinusoidal trick

The paper builds each position's vector from sine and cosine waves at geometrically spaced frequencies — dimension 0 oscillates fast, the last dimensions oscillate over thousands of positions:

Why waves, of all things? Three properties, no training required:

• Unique: each position gets a distinct pattern across dimensions — like a binary counter, but smooth: fast bits flip every step, slow bits flip rarely.
• Relative-friendly: thanks to trig identities, the encoding of position pos+k is a fixed linear transform of position pos. "k tokens apart" looks the same everywhere in the sequence — easy for heads to learn.
• Extends past training: the formula yields a vector for any position, even longer than any training sequence.

One footnote for modern context: this exact sinusoidal scheme is the 2017 design. Later models often use learned position embeddings (GPT-2) or rotary embeddings (RoPE, in Llama-family models) — different recipes, same job: make order visible to an order-blind mechanism.

You now own all the parts. Time to assemble Figure 1 — the famous architecture diagram. Remember the paper's original job: machine translation (English → German). That task shapes everything: you need to read one sequence and write another.

Two stacks, two jobs

• The encoder reads the entire source sentence and produces one context-rich vector per source token. It's a comprehension pipeline — note: per-token outputs, not one squashed summary. The RNN bottleneck is gone.
• The decoder writes the output one token at a time, and at every step it can consult both what it has written so far and the encoder's reading of the source.

Inside one layer

Each stack is N = 6 identical layers. An encoder layer has two sub-blocks: multi-head self-attention (tokens exchange context) then a feed-forward network (FFN) — a small two-layer MLP applied to each token independently. A useful mental split: attention moves information between tokens; the FFN processes information within each token.

A decoder layer has three sub-blocks: masked self-attention over the output written so far (masked why? Module 8), then cross-attention, then an FFN. Cross-attention is the bridge between the stacks — and it's just the Module 4 recipe with mixed participants: queries come from the decoder, keys and values come from the encoder's output. The token being written asks, "which source words are relevant to me right now?"

The glue: residuals and layer norm

Around every sub-block, two stabilizers (the paper's "Add & Norm"):

• Residual connection: output = x + SubBlock(x) — each block adds a refinement to its input rather than replacing it. Information (like positional encodings) survives all the way up, and gradients flow down a clean highway through deep stacks.
• Layer normalization: rescales each token's vector to a standard distribution. Numerical hygiene, like the √d_k trick.

You only need the halves you need

The two stacks proved useful on their own — this is the part that connects 2017 to today:

So when you hear that modern LLMs are "decoder-only Transformers": it's the right tower of this diagram, minus cross-attention, scaled up by several orders of magnitude.

Module 7 left a thread hanging: the decoder's self-attention is masked. This small detail is what lets Transformers train efficiently, so it deserves its own module.

The problem: training would let the model cheat

At inference time the decoder generates left to right; token 5 literally doesn't exist when token 4 is being computed. But during training we already have the full correct output sentence, and we'd love to compute the loss for all positions in one parallel pass (that was the whole point of ditching RNNs!).

Feed the full target sentence into unrestricted self-attention, though, and position 4's prediction of token 5 can just attend to token 5 sitting right there in the input. The model learns to copy, not predict. Useless.

The fix: set forbidden scores to −∞

Recall the n×n score matrix from QKᵀ: row i = token i's scores over every position. The causal mask (a.k.a. look-ahead mask) overwrites every score where column > row — every "look into the future" — with −∞ before the softmax:

scores[i][j] = (j <= i) ? scores[i][j] : -Infinity;
weights = softmax(scores);   // e^(-∞) = 0 → future weights are exactly 0

Why −∞ and not just 0? Because softmax exponentiates: a score of 0 still yields a positive weight (e⁰ = 1). Only −∞ guarantees a weight of exactly zero. The surviving weights re-normalize over the allowed positions and still sum to 1.

The other mask: padding

A humbler mask does bookkeeping. GPUs want rectangular batches, but sentences have different lengths, so short ones are padded with a dummy [PAD] token. A padding mask sets scores toward pad positions to −∞ so no real token wastes attention on filler. Same −∞ trick, different policy.

Training: the world's largest autocomplete exam

How do all those W matrices get good values? Astonishingly simply. Take mountains of text. For every position, the model predicts a probability distribution over the next token; the loss (cross-entropy) punishes it for assigning low probability to the token that actually came next. Gradient descent nudges every weight to do slightly better. Repeat ~trillions of times.

No labels, no grammar rules, no knowledge base — the text is its own answer key (this is what "self-supervised" means). To get good at next-token prediction at scale, the model is forced to internalize syntax, facts, and reasoning patterns, because all of those reduce prediction error. The masked parallel training from Module 8 is what makes this computationally feasible.

Inference: the autoregressive loop

Generation is the training objective run in a loop:

while (!done) {
  logits = transformer(tokens);          // one score per vocab entry, for the last position
  next   = sample(softmax(logits));      // pick a token (temperature lives here)
  tokens.push(next);                     // the output becomes input
}

That's an LLM "thinking": predict one token, append it, predict again. Each new token gets full attention over everything before it — your prompt and the answer so far.

Your map for reading the actual paper

You're ready. Here's the paper, section by section, in your new vocabulary — arxiv.org/abs/1706.03762:

Two reading tips: Figure 1 is Module 7's diagram — keep them side by side. And when notation gets dense in §3.2, translate to the soft-database story: queries search keys, softmax weights, blend values.

Where to go next

• The Illustrated Transformer (Jay Alammar) — the classic visual walkthrough; great second pass.
• “Let's build GPT” (Andrej Karpathy, YouTube) — implement a decoder-only Transformer from scratch in ~2 hours of Python. The ultimate test of understanding.
• nanoGPT (GitHub) — the same model as a tiny readable codebase; read model.py and recognize every line.