Positional Encoding and the Transformer Block

Attention has a blind spot, and it is a big one. The mechanism you have built over the last two lessons has no idea what order your words are in.

This lesson fixes that with positional encoding, then zooms out to show you the full transformer block: the repeating unit that attention actually lives inside. By the end you will have the complete architecture in your head, from your prompt going in to a prediction coming out.

What You'll Learn

• Why attention is blind to word order, and why that is a real problem
• How positional information gets added back into the model
• The other pieces of a transformer block besides attention
• What residual connections and normalization do, in plain language
• Why stacking the same block dozens of times produces depth of understanding

Attention cannot see order

Look again at what attention does. Every token's query is scored against every token's key, the scores are turned into weights, and values are blended. Notice what is missing from that description: nowhere does the position of a token enter the calculation.

The consequence is stark. To a bare attention layer, these two sentences are identical:

The dog bit the man.

The man bit the dog.

Same tokens, same set of pairwise scores, same blended result. A mechanism that treats a sentence as an unordered bag of words is useless for language, where order carries an enormous amount of the meaning.

This is the price of dropping recurrence. The old sequential models got word order for free, because they physically read left to right. Transformers gave that up in exchange for parallelism, so order has to be put back deliberately.

Putting position back in

The solution is to add position information into the token representations before attention ever sees them. Each position in the sequence gets its own distinctive pattern of numbers, and that pattern is combined with the token's embedding.

The result is that the word "dog" in position 2 and the word "dog" in position 5 enter attention as slightly different things. Attention still does not know about order explicitly, but the numbers it receives now carry the information, so it can learn to use it.

A good way to picture it: the embedding is a book's subject and the position signal is its shelf number. Attention reads both at once, so it can consider not only what a token means but where it sits relative to everything else.

Two details worth knowing, without going near the math.

The pattern is designed to express relative distance. The original transformer used a set of overlapping wave patterns of different frequencies, chosen so the model could work out how far apart two positions are, not just their absolute numbers. Relative distance is what language actually cares about. "The word three back" matters; "the word at index 847" rarely does.

Modern models mostly use a newer scheme. Today's large models typically use rotary position embeddings, usually written RoPE, which bakes position into the query and key comparison itself rather than adding it at the input. The goal is the same and the intuition is unchanged. RoPE is simply better behaved when a model has to handle sequences longer than the ones it trained on, which is one reason context windows have grown so much. If you see "RoPE scaling" in a model's release notes, that is a team stretching the position scheme to support a longer context.

The transformer block

Attention is not the whole model. It is one component inside a repeating unit called a transformer block, and each block has four parts.

Multi-head attention is the part you already know: each token gathers relevant information from the other tokens.

The feed-forward network is the part people skip, and it deserves better. After attention has gathered context, every token is passed independently through a small two-layer network. No token talks to any other during this step. This is where the model does its per-token "thinking": recognizing what this particular gathered mixture represents and transforming it accordingly. Worth knowing: the feed-forward layers usually hold the majority of a model's parameters, considerably more than attention does. When you read that a model has 70 billion parameters, most of them are here.

So the block alternates between two modes. Attention mixes information across tokens. The feed-forward network processes each token on its own. Gather, then think. Gather, then think.

Residual connections are the small addition arrows in the diagram. After each part, the input to that part is added back to its output. The effect is that each layer proposes an adjustment to the representation rather than replacing it, so information can flow all the way through a deep stack without being destroyed. Without residuals, models this deep are effectively untrainable.

Normalization rescales the numbers to a consistent range before each part. It is plumbing rather than intelligence, but it is what keeps values from exploding or collapsing as they pass through dozens of layers.

Stacking blocks is where depth comes from

One block is not a model. A real transformer stacks the same block over and over: a small model might use 12, a mid-size one 32, a frontier model considerably more. The blocks are identical in shape and completely different in what they have learned.

What emerges from stacking is a rough progression. Early layers tend to handle surface-level patterns: which token follows which, basic grammatical agreement. Middle layers capture more structure and relationships. Later layers work with more abstract, task-relevant properties. Each layer's attention operates on representations that previous layers have already enriched, so by the top of the stack a token's representation reflects the whole prompt filtered through many rounds of gathering and processing.

After the final block, one last step converts the representation of the most recent token into a probability for every possible next token, and the model picks one. That is the handoff into next-token prediction, covered in How LLMs Actually Work.

Putting it all together

Here is the entire path your prompt takes:

And then the whole thing runs again for the next token, and the next, until the answer is complete.

Key Takeaways

• Attention is blind to word order on its own, so "the dog bit the man" and "the man bit the dog" would look identical without a fix.
• Positional encoding adds a distinctive per-position signal to each token's embedding before attention, letting the model work with relative distance; modern models typically use RoPE, which is part of why context windows have grown.
• A transformer block pairs multi-head attention, which mixes information across tokens, with a feed-forward network, which processes each token independently and holds most of the model's parameters.
• Residual connections let each layer adjust rather than replace the representation, and normalization keeps the numbers stable, which together make deep stacks trainable.
• Stacking dozens of identical blocks produces the depth: early layers handle surface patterns, later layers work with more abstract structure, and the final layer hands off to next-token prediction.