The Memory Wall: Why Memory Limits AI
When people compare AI hardware, they often focus on raw speed. In practice, the thing that most often decides which model you can run and how fast it answers is memory: how much fast memory sits next to the chip, and how quickly it can feed data to the chip.
Engineers call this the memory wall. Processors have become fast much more quickly than memory has, so chips often wait for data. This lesson explains why that matters so much for AI, with simple arithmetic you can do yourself.
What You'll Learn
- Why a model's weights must fit in fast memory
- How to estimate how much memory a model needs
- Why memory bandwidth sets the speed of writing an answer
- Why long conversations need extra memory
- How engineers work around the memory wall
The weights must fit
To answer quickly, a model's weights need to sit in the fast memory right next to the processor. On a GPU, this is its own dedicated memory. On many laptops and phones, the processor and AI chip share one pool of memory.
If the weights do not fit, the system has to keep swapping parts of the model in from slower memory or storage. That can make answers painfully slow, or stop the model from running at all.
Estimate the memory yourself
You can roughly estimate how much memory a model's weights need with one multiplication:
number of parameters × bytes per parameter
Each parameter is a number, and numbers take space. At a common full-quality setting, each parameter uses 2 bytes.
Weights alone. Running the model needs some extra memory on top.
| Criteria | Parameters | Memory at 2 bytes each |
|---|---|---|
| Small model | 1 billion | About 2 GB |
| Medium model | 8 billion | About 16 GB |
| Large model | 70 billion | About 140 GB |
Parameters
- Small model
- 1 billion
- Medium model
- 8 billion
- Large model
- 70 billion
Memory at 2 bytes each
- Small model
- About 2 GB
- Medium model
- About 16 GB
- Large model
- About 140 GB
This simple sum explains a lot:
- A small model can fit on a phone or ordinary laptop.
- A medium model needs a laptop with plenty of memory or a good GPU.
- A large model needs more memory than any single consumer device has, so it has to be split across several data center chips.
The next lesson shows how quantization shrinks these numbers, which is how larger models end up running on everyday hardware.
Bandwidth sets the speed
Fitting the model is only step one. Speed depends on memory bandwidth: how much data can move from memory to the processor each second.
Recall from lesson 1 that a language model writes one token at a time, and for each token it has to read through a large share of its weights. So for each new token, a lot of data must flow from memory to the chip.
That leads to a useful rule of thumb:
Tokens per second is limited by memory bandwidth ÷ size of the weights read per token.
- Need next tokenThe model must run again
- Read the weightsLarge amounts of data move from memory
- CalculateThe chip is fast, this part is quick
- Write one tokenThen do it all again
In plain terms: a bigger model writes more slowly on the same hardware, because there is more to read for every token. And hardware with faster memory writes faster, even if its raw calculation speed is similar. This is why AI data center chips are paired with extremely fast, expensive memory stacked right next to the processor.
Some models use a design called mixture of experts, where only part of the model's weights are used for each token. That reduces how much data must be read per token, which helps speed. How LLMs Actually Work explains mixture of experts in more detail.
Long conversations need extra memory
The weights are not the only thing in memory. While a model works through a conversation, it keeps saved intermediate results for every token so far, so it does not have to recalculate them each time. This is often called the KV cache. You do not need the name, only the effect:
- The longer the conversation or document, the more memory this takes.
- Very long contexts can need a lot of memory on top of the weights.
- Serving many users at once multiplies this, since each conversation needs its own.
This is one of the physical reasons behind context-window limits, and why very long contexts are harder and slower to serve.
Working around the wall
Engineers attack the memory wall from several directions:
- Faster memory placed as close to the chip as possible.
- Shrinking the numbers so there is less data to store and move (next lesson).
- Splitting a model across many chips, each holding part of it (final lesson).
- Batching: answering many users' requests together, so each read of the weights serves several people at once. This is a big reason serving AI at scale is more efficient than it first appears.
- Smarter model designs like mixture of experts that read less data per token.
Key Takeaways
- The memory wall: chips got fast faster than memory did, so AI often waits on data.
- A model's weights must fit in fast memory. Estimate it as parameters × bytes per parameter.
- Memory bandwidth limits how many tokens per second a model can write. Bigger models write more slowly on the same hardware.
- Long conversations need extra memory for saved intermediate results, which grows with context length.
- Engineers push back with faster memory, smaller numbers, splitting models across chips, batching, and smarter designs.