Why Transformers Run in Parallel, and What Long Prompts Cost

You now have the full architecture. This final lesson cashes it in for things you can actually use: why transformers could be trained at a scale recurrent models never reached, why doubling your prompt more than doubles the work, and why the first word of a response takes longer to arrive than the rest.

Everything here follows directly from the mechanism you have already learned. No new machinery, just consequences.

What You'll Learn

• Why attention parallelizes and recurrence does not
• Why attention cost grows with the square of the prompt length
• What that quadratic growth means for long documents and long chats
• Why the first token of an answer is slow and the rest are fast
• Practical habits that follow from how the mechanism works

Why attention parallelizes

A recurrent model could not process word five until it had finished word four, because word five needed the memory state that word four produced. That dependency chain is unavoidable and it is the reason those models trained slowly. A GPU has thousands of cores, and a sequential chain uses a handful of them at a time.

Attention has no such chain. Every query-key score is independent of every other. The score between token 1 and token 7 does not need the score between token 6 and token 7 to exist first. So all of the scores can be computed simultaneously, as one large matrix operation, which is precisely the shape of work a GPU is built for.

This is the underrated reason transformers won. The architecture was not just more accurate. It was trainable at a scale nothing before it could reach, because it could actually use the hardware. Scale then delivered the capabilities everyone noticed.

The catch: cost grows with the square

Parallel does not mean free. Look at what attention computes: every token is scored against every other token. That is a full grid.

• 10 tokens produce 100 pairs
• 100 tokens produce 10,000 pairs
• 1,000 tokens produce 1,000,000 pairs
• 10,000 tokens produce 100,000,000 pairs

Double the length and you get four times the pairs. Ten times the length gives you a hundred times the pairs. This is what people mean when they say attention is quadratic in sequence length.

This single fact explains a lot of what you observe when using AI tools.

Why long context windows were hard to build. Going from an 8,000-token context to 128,000 is not sixteen times the work for the attention step, it is roughly 256 times. Getting there took real engineering: memory-efficient attention implementations, position schemes like RoPE that stretch gracefully, and architectural variants that avoid scoring every single pair. Long context is an achievement, not a free setting.

Why very long chats slow down and drift. Every turn resends the whole conversation, so the sequence keeps growing and the attention grid keeps growing faster. Attention is also a fixed budget spread over more and more tokens, so any individual instruction gets a thinner slice. Starting a fresh chat for a new task is not superstition; it is giving your instruction less competition.

Why pasting an enormous document has a real price. The tokens are billed, as you know from tokenization, and the attention work on them grows faster than their count. Pasting the relevant three pages instead of the whole 200-page report is meaningfully cheaper and usually produces a better answer.

Note the honest nuance: for typical prompt sizes today, the quadratic attention step is not always the single dominant cost, since the feed-forward layers are large and grow only linearly. But as sequences get long, the quadratic term is the one that takes over, and it is why the frontier of context length has been so hard-won.

Why the first token is slow

Watch an assistant carefully and you will notice a pause before the first word appears, then a smooth stream of words afterward. That is two genuinely different phases of work.

Prefill is processing your entire prompt. Every token in your prompt is embedded, run through every transformer block, and its keys and values are computed and stored. This phase is fully parallel, so it uses the hardware well, but there is a lot of it, and it is the quadratic part. A long prompt means a long pause.

Decode is generating the answer, one token at a time. Here the model genuinely cannot parallelize, because it has to know what word it just chose before choosing the next one. But each step is small: the new token attends to the stored keys and values from everything before it, which were already computed during prefill and are cached.

That cache of stored keys and values is called the KV cache, and it is why generation stays fast: without it the model would reprocess the entire conversation for every single word it writes. It also takes real memory, growing with the conversation length, which is another reason very long sessions get expensive to serve.

This also explains a pricing detail you may have noticed. Input tokens are usually cheaper per token than output tokens, even though there are more of them. Input is processed in one efficient parallel pass; output requires a separate sequential pass through the entire model for every token produced.

What this means for how you work

Nothing here is a trick. These are just habits that follow from the mechanism.

• Send what is relevant, not everything. Attention cost grows faster than length, and a thinner budget spread across less material gives your actual question more weight.
• Start a new chat when the task changes. A fresh conversation is a smaller sequence and a cleaner attention budget.
• Put the instruction where it can win. A clear ask near the content it applies to competes against less than one buried in the middle of a wall of text.
• Ask for the length you want. Output is the sequential phase, so a shorter answer is genuinely faster and cheaper, not just tidier.
• Read model specs with context. Context window size is a real engineering achievement, and using all of it always has a cost.

Where to go next

You now understand the mechanism behind every modern AI model: how tokens gather context from each other, why many heads run at once, how word order is preserved, and what it costs to run.

Two natural next steps. How LLMs Actually Work covers the layers around this one: tokens, context windows, parameter counts, Mixture of Experts, and inference cost. Prompt Engineering turns the intuition you just built into better everyday results.

Key Takeaways

• Attention parallelizes because every token pair is scored independently, which let transformers use GPU hardware fully and train at a scale recurrent models never reached.
• Attention cost is quadratic: double the tokens and you quadruple the pairs, which is why long context windows were hard to build and why long chats get slower and more expensive.
• Responses have two phases: prefill processes your whole prompt in parallel and causes the initial pause, while decode generates one token at a time and streams out.
• The KV cache stores keys and values from earlier tokens so generation stays fast, which is also why input tokens are usually cheaper than output tokens.
• Practical habits follow directly: send only relevant context, start fresh chats for new tasks, place instructions where they stand out, and ask for the length you actually want.