Understanding Context Windows
Context windows are the memory limit for AI models. Understanding how they work is essential for building chains that don't break due to context overflow.
What is a Context Window?
The context window is the total amount of text a model can "see" at once. This includes:
- Your system prompt
- Conversation history
- The current user message
- Any injected context (documents, previous outputs)
- The model's response
┌─────────────────────────────────────────┐
│ CONTEXT WINDOW │
│ (e.g., 128k tokens) │
├─────────────────────────────────────────┤
│ System Prompt │ 1,000 │
│ Chain State │ 5,000 │
│ Previous Step Outputs │ 15,000 │
│ Current Step Input │ 3,000 │
│ ───────────────────────────────────── │
│ USED │ 24,000 │
│ AVAILABLE FOR RESPONSE │ 104,000 │
└─────────────────────────────────────────┘
Token Basics
Tokens are the units models use to measure text:
- Average: ~4 characters per token in English
- Words: Most common words are 1-2 tokens
- Code: Often uses more tokens due to special characters
- JSON: Structure overhead adds tokens
Context Window Sizes
Different models have different limits:
| Model | Context Window |
|---|---|
| GPT-3.5 | 4k - 16k tokens |
| GPT-4 | 8k - 128k tokens |
| Claude 3 | 200k tokens |
| Gemini | 32k - 1M tokens |
Important: Larger isn't always better. Longer contexts can:
- Increase latency
- Increase cost
- Sometimes decrease quality (lost in the middle)
Context in Chains
Each step in a chain needs context. This accumulates:
Step 1: Input (1000 tokens)
└→ Output (500 tokens)
Step 2: Input (1000) + Step1 Output (500) = 1500 tokens
└→ Output (800 tokens)
Step 3: Input (1000) + Step1 (500) + Step2 (800) = 2300 tokens
└→ Output (1000 tokens)
By Step 10: Could easily exceed 10,000+ tokens
The "Lost in the Middle" Problem
Models don't attend equally to all parts of context:
Attention strength:
███████████░░░░░░░░░░░░░░░░░███████████
^ ^
Beginning Middle End
(Strong) (Weak) (Strong)
This means:
- Put important information at the start and end
- Critical instructions should be in the system prompt
- The most recent context matters most
Managing Context in Chains
Strategy 1: Minimal Context
Only pass what's absolutely needed:
// Bad: Pass everything
const step2Input = {
originalDocument: fullDocument, // 5000 tokens
step1Analysis: completeAnalysis, // 2000 tokens
metadata: allMetadata // 500 tokens
};
// Good: Pass only what's needed
const step2Input = {
summary: step1Analysis.summary, // 200 tokens
keyEntities: step1Analysis.entities, // 100 tokens
relevantMetadata: { // 50 tokens
documentType: metadata.type
}
};
Strategy 2: Context Budgets
Allocate tokens per step:
const contextBudget = {
systemPrompt: 500,
originalInput: 1000,
previousSteps: 2000,
currentStep: 500,
responseBuffer: 2000,
total: 6000
};
function checkBudget(content, allocation) {
const tokens = estimateTokens(content);
if (tokens > allocation) {
return summarize(content, allocation);
}
return content;
}
Strategy 3: Rolling Window
Keep only recent context:
class RollingContextWindow {
constructor(maxTokens) {
this.maxTokens = maxTokens;
this.items = [];
}
add(item) {
this.items.push(item);
// Remove oldest items if over budget
while (this.getTotalTokens() > this.maxTokens) {
this.items.shift();
}
}
getTotalTokens() {
return this.items.reduce((sum, item) => sum + item.tokens, 0);
}
}
Measuring Context Usage
Token Estimation
Monitoring Token Usage
async function trackedStep(prompt, input) {
const inputTokens = estimateTokens(prompt + JSON.stringify(input));
const startTime = Date.now();
const result = await runPrompt(prompt, input);
const duration = Date.now() - startTime;
const outputTokens = estimateTokens(result);
return {
result,
metrics: {
inputTokens,
outputTokens,
totalTokens: inputTokens + outputTokens,
duration,
tokensPerSecond: outputTokens / (duration / 1000)
}
};
}
Exercise: Optimize Context Usage
Review this chain state and optimize it:
Key Takeaways
- Context windows limit how much text a model can process
- Tokens are the unit of measurement (~4 chars per token)
- Context accumulates across chain steps
- Models attend more to beginning and end of context
- Pass only necessary context between steps
- Use rolling windows or summaries for long chains
- Monitor token usage to prevent overflow
- Budget tokens across chain steps
Next, we'll explore strategies for accumulating context effectively.
التالي · الدرس 23 من 43
Context Accumulation Strategies
قراءة 3 دقيقة
أنهيت 22 من 43 درسًا

اسأل Byte عن هذا الدرس
محادثة خاصة حول هذا الدرس. قرأ Byte الدرس كاملًا، ويجيب خلال ثوانٍ، ويُظهر القسم الذي تأتي منه كل إجابة. يمكنك السؤال بأي لغة.
الأسئلة والأجوبة
أسئلة عامة حول هذا الدرس. يجيب AI Tutor فورًا بلغتك، ويراجع فريق FreeAcademy الإجابات، ويمكن لكل المتعلمين قراءتها.

