The Context Window: The Model's Working Memory

You have probably had this experience: a long chat with an AI starts sharp, then slowly drifts. It forgets a detail you gave it earlier, contradicts itself, or loses the thread. It can feel like the model is getting tired. It is not. You are running into the context window, the hard limit on how much text a model can hold in mind at once.

This lesson explains what the context window is, what fits inside it, what falls out, and why long conversations degrade. It is one of the most practical things you can understand as a daily AI user, because it changes how you structure your prompts and your chats.

What You'll Learn

• What the context window is and how it is measured
• What counts against it (hint: everything)
• Why long chats start to forget and drift
• Simple habits that keep the model on track

Context is the model's short-term memory

The context window is the maximum number of tokens the model can consider at one time. It includes everything in the current exchange:

• Any hidden system instructions the app adds
• Your entire conversation so far, both your messages and the model's replies
• Any files, documents, or data you have pasted or attached
• The new response the model is about to generate

Think of it as a whiteboard of a fixed size. The model can read anything written on the whiteboard right now, and it uses all of it to predict the next token. But the whiteboard has edges. Once it is full, something has to be erased to make room.

Crucially, a base model has no memory between separate conversations. When you start a new chat, the whiteboard is wiped clean. Anything the model seems to "remember" across sessions comes from an extra feature the app bolts on (a saved-memory or profile system), not from the model itself. Within a single chat, the context window is the only memory there is.

Everything counts against the window

A common misconception is that only your latest message matters. In reality, most chat apps resend the entire conversation to the model on every turn, because the model is stateless and needs the whole history re-supplied to stay coherent. That is how it "remembers" what you said three messages ago: the app keeps handing it the transcript.

This has a direct consequence. A long conversation is not free. Each new turn re-sends everything before it, so both your token usage and the pressure on the context window grow as the chat gets longer. Pasting a giant document early in a chat means that document is (in effect) re-read on every subsequent turn until it scrolls out of the window.

What happens when the window fills up

Context windows are large in modern models, but they are still finite. When a conversation exceeds the limit, the app has to drop something, and the usual casualty is the oldest text. That is the mechanism behind the classic long-chat failures:

• The model forgets a name or constraint you set at the very start, because it literally scrolled off the whiteboard.
• It contradicts an earlier decision, because the earlier decision is no longer in view.
• It repeats questions you already answered.

There is a second, subtler effect. Even when text technically still fits, models tend to pay the most attention to the beginning and the end of a long context, and can gloss over material buried in the middle. So a crucial instruction sandwiched in the middle of a very long prompt may be underweighted. This is sometimes called the "lost in the middle" effect. The lesson: placement matters, not just whether something fits.

Habits that keep the model on track

You do not need a bigger model to fix drift. A few simple habits go a long way:

• Start a fresh chat for a new task. A clean window removes irrelevant history that could confuse the model and wastes no tokens on it. This is the single most effective fix.
• Restate key constraints late. If a detail really matters, repeat it near the end of your prompt, where the model attends most, rather than relying on something you said far earlier.
• Summarize, then continue. In a long, valuable chat, ask the model to summarize the important decisions so far, then start a new chat and paste that summary in. You keep the signal and drop the noise.
• Do not dump everything. Paste only the parts of a document that matter. More context is not automatically better; irrelevant text dilutes attention and eats the window.
• Put instructions where they will be seen. For long inputs, place your actual request at the top or bottom, not buried in the middle.

A note on "bigger context" claims

Model makers advertise ever-larger context windows, and bigger genuinely helps for tasks like analyzing a long report in one go. But a large window is not the same as perfect recall. Filling a huge context to the brim can still produce the middle-neglect problem and costs more tokens per turn. Treating context as a scarce, well-organized resource beats treating it as an infinite bucket, no matter how large the advertised number is.

If you want to go deeper on how providers describe and price context, the FreeAcademy blog post "LLM Context Windows Explained" covers the current landscape.

Key Takeaways

• The context window is the maximum tokens a model can consider at once, and it includes the full conversation, not just your last message.
• Models are stateless between chats; within a chat, the resent transcript is the only memory, and a saved-memory feature is a separate add-on.
• When the window overflows, the oldest text is dropped, which is why long chats forget early details and drift.
• Even within the limit, models attend most to the start and end, so key instructions buried in the middle can be missed.
• Start fresh chats, restate key constraints late, summarize-and-continue, and paste only what matters to keep the model accurate.