Shrinking Models: How Quantization Works

The previous lesson showed that memory is often the limit on which model you can run and how fast it answers. The most common way around that limit is surprisingly simple: store each number with less precision. This is called quantization, and it is the main reason capable AI models can run on laptops and phones.

This lesson explains what quantization does to the numbers inside a model, why models can lose so much precision and still work well, and where the trade-offs start to show.

What You'll Learn

• What precision means for the numbers in a model
• How quantization shrinks a model, with simple arithmetic
• Why models tolerate lower precision better than you might expect
• Where quality starts to drop, and what to watch for
• How to read common labels like 16-bit, 8-bit, and 4-bit

Precision: how exactly a number is stored

Every weight in a model is a number, such as 0.0371 or -1.28. Computers store numbers using a fixed number of bits. More bits means the number can be stored more exactly. Fewer bits means it has to be rounded.

A simple comparison: a price stored as 12.3749 is very exact. Stored as 12.37, it is close. Stored as 12, it is rougher, but for many purposes still useful.

Models are often trained with numbers stored in 16 bits each (2 bytes). Quantization converts them to fewer bits, commonly 8 bits (1 byte) or 4 bits (half a byte).

How much it saves

Use the same sum from the previous lesson: parameters × bytes per parameter.

Two benefits come together:

1. It fits in less memory, so it can run on smaller devices.
2. It runs faster, because there is less data to read from memory for every token. Recall that memory bandwidth, not calculation, is often the speed limit.

Why models can tolerate it

It seems surprising that rounding billions of numbers would not break a model. A few reasons it usually does not:

• Knowledge is spread out. A model does not store a fact in one precise number. It is spread across many weights, so small rounding errors in each one tend to average out.
• Most weights are small and similar. The range of values is narrow enough that fewer bits can still represent them well.
• Smart rounding. Quantization methods do not just chop off digits. They group weights, choose scales carefully, and keep the most sensitive parts at higher precision.

Where the trade-offs show

Quantization is not free. As precision drops, some losses start to appear:

• Subtle quality loss. Answers may be slightly less accurate, less fluent, or less consistent, often in ways that are hard to notice in casual use.
• Harder tasks suffer first. Careful multi-step reasoning, math, code, and less common languages tend to show the loss before simple chat does.
• Smaller models feel it more. A large model has more room to absorb rounding than a small one.
• Very low precision breaks down. Below about 4 bits, quality usually drops quickly for most models.

Reading the labels

If you ever download a model to run yourself, you will see labels that describe quantization, such as "Q4," "Q8," "4-bit," "int8," or "fp16." They all describe how many bits each number uses:

• fp16 / bf16: 16-bit, full or near-full quality.
• 8-bit / int8 / Q8: half the size, usually very close in quality.
• 4-bit / Q4: a quarter of the size, the most common choice for running models on personal devices.

Run AI on Your Own Computer shows how to pick and run a quantized model step by step. If you are deciding whether to run AI locally at all, the blog post Local LLMs vs Cloud LLMs compares the options.

Other ways to shrink models

Quantization is the most common technique, but not the only one:

• Distillation: training a smaller model to imitate a larger one, so it learns much of its behavior in fewer parameters.
• Pruning: removing weights that contribute little.
• Smaller designs: models built from the start to be efficient on phones and laptops.

These are often combined. A small, distilled model, quantized to 4 bits, is how many on-device AI features work.

Key Takeaways

• Quantization stores each weight with fewer bits, trading a little precision for a lot of space.
• Going from 16-bit to 8-bit halves memory, and 4-bit quarters it, which also speeds up answers.
• Models tolerate it because knowledge is spread across many weights and rounding errors tend to average out.
• Losses show first on hard reasoning, math, code, and smaller models, and quality falls fast below about 4 bits.
• Labels like fp16, int8, Q8, and Q4 describe precision. Distillation and pruning are other ways to shrink models.