Query, Key, and Value: How Self-Attention Works

Last lesson you saw why attention exists: every token needs to consult the other tokens around it to know what it means. This lesson is the heart of the course. You will see how the model actually decides which tokens to consult, using three ideas with unhelpful names: query, key, and value.

The names come from database search, and that is the fastest way to understand them. Once you have the analogy, the whole mechanism collapses into something you can explain at dinner.

Still no math. You will see the shape of the computation, not the formula.

What You'll Learn

• What query, key, and value mean, using a library search analogy
• How a model scores which tokens are relevant to which
• What an "attention weight" is and why the weights always add up to 1
• How the weighted blend produces a context-aware version of each token
• Why this is called self-attention

The library analogy

Imagine you walk into a library with a question in your head. Three things are in play.

Your query is the question you are holding: "I need something about cooking with cast iron."

Every book on the shelf has a key: the spine label, the title, the subject tag. It is the advertisement of what that book is about, the thing you scan to decide whether the book is relevant to you.

Every book also has value: the actual content inside. That is what you walk away with if you decide the book is worth reading.

You match your query against every key, decide how relevant each book is, and then take content from the books in proportion to how relevant they were. A book that matched perfectly contributes a lot. A book about tax law contributes almost nothing.

Self-attention does exactly this, except every token is both a reader and a book. Each token produces a query (what am I looking for?), a key (what do I offer?), and a value (what I contribute if chosen).

Where the three come from

Each token arrives at the attention layer as a list of numbers, its embedding, which encodes what that token generally means. The model multiplies that list by three different sets of learned weights to produce three new lists: the query, the key, and the value for that token.

The important word is learned. Nobody hand-wrote rules like "pronouns should look for nouns." During training, the model adjusted those three sets of weights until the queries and keys it produced made its next-token predictions accurate. The behavior you see, pronouns reliably attending to the nouns they refer to, is a pattern that emerged from that pressure.

So for a five-token prompt, the model produces five queries, five keys, and five values. Everything that follows is comparison and blending.

Step one: score every pair

For a given token, the model compares its query against the key of every token in the sequence, including its own. Each comparison produces a single number: a raw score for how well that query matches that key.

Take the sentence "The cat sat because it was tired."

The token "it" produces a query that, loosely, means I am a pronoun and I need to know which thing I refer to. That query is scored against every key:

• against the key for "cat": high score
• against the key for "sat": low score
• against the key for "because": low score
• against the key for "tired": medium score
• against the key for "The": very low score

The model does this for every token, not just "it". So a sentence of five tokens produces a full five by five grid of scores. This grid is what people mean when they show you a colorful attention map: rows are the token doing the looking, columns are the tokens being looked at, and brightness is the score.

Step two: turn scores into weights

Raw scores are awkward to work with. They can be any size, positive or negative. So the model pushes each row of scores through a step called softmax, which does two useful things: it makes everything positive, and it makes each row sum to exactly 1.

The result is a set of attention weights. For our example, the row for "it" might come out roughly like this:

Token being looked at	Attention weight
The	0.02
cat	0.71
sat	0.06
because	0.04
it	0.09
tired	0.08

Read that as a budget. The token "it" has one full unit of attention to spend, and it spends 71 percent of it on "cat". This is the model saying, in the only language it has, "it" means the cat.

Because the weights must add to 1, attention is always a trade-off. Paying more attention to one token necessarily means paying less to another. There is no way for a token to simply attend hard to everything.

Step three: blend the values

Now the payoff. The model takes the value of every token, multiplies each by its attention weight, and adds them all together. The output is a new list of numbers for the token "it", one that is mostly made of the cat's value, with small contributions from the rest.

That new list is the point of the whole exercise. The token "it" entered the layer as a generic pronoun that could mean anything. It leaves carrying the meaning of "cat". The representation has become contextual.

Run the same three steps for every token in parallel, and every token in your prompt comes out the other side enriched by whatever else in the prompt was relevant to it.

Why "self"-attention

It is called self-attention because the queries, keys, and values all come from the same sequence. The sentence is looking at itself. Each token compares itself to its own neighbors rather than to some separate external document.

There is a related variant called cross-attention, where the queries come from one sequence and the keys and values come from another. That is how translation models let the output sentence consult the input sentence, and how image models let generated pixels consult a text prompt. Same machinery, different sources. For understanding chatbots, self-attention is the one that matters.

One practical consequence

Attention weights are a budget, and your prompt sets the terms. When you write a long, rambling prompt with a single important instruction buried in the middle, that instruction is competing for weight against everything else you wrote. When you write a short prompt where the instruction sits right next to the task, it faces far less competition.

This is a mechanical reason behind advice you have probably already heard: put important instructions where they stand out, remove filler, and do not bury the ask. You are not being polite to the model. You are shaping a budget it has no choice but to spend.

Key Takeaways

• Each token produces three things from learned weights: a query (what it is looking for), a key (what it offers), and a value (what it contributes).
• The model scores every query against every key, producing a grid of relevance numbers, the thing visualized in attention maps.
• Softmax turns each row of scores into attention weights that are positive and sum to 1, making attention a fixed budget that must be traded off.
• The output for each token is a weighted blend of values, which turns a generic token into a context-aware one, so "it" comes out carrying the meaning of "cat".
• It is called self-attention because query, key, and value all come from the same sequence; cross-attention is the variant that consults a different one.