The Overlord's Lexicon
What is Quantization?
Also known as: Model quantization, Quantisation
Quantization is a technique that shrinks an AI model by storing its numbers (weights) at lower precision — for example 8-bit or 4-bit instead of 16-bit — so it uses less memory and runs faster, usually with only a small loss in quality.
Quantization is how you fit a big AI model into a small machine. A model is essentially billions of numbers — its parameters. Store each one with less precision and the whole model takes up far less memory and runs faster.
How it works
Models are commonly trained and shipped with 16-bit numbers. Quantization maps those values onto a smaller set of levels:
- 16-bit → 8-bit roughly halves memory.
- 16-bit → 4-bit cuts it to about a quarter.
Some quick maths: a 7-billion-parameter model at 16 bits needs roughly 14 GB just for its weights. At 4 bits, roughly 3.5 GB — small enough for many laptops. Clever methods (GPTQ, AWQ, and the k-quant schemes used in GGUF files, among others) choose how to round values so that the most important information survives.
Quantization can happen after training (post-training quantization, the common approach for running open models locally) or be built into training (quantization-aware training) for better quality at low precision.
An analogy for the meatbags
Quantization is like converting a lossless studio audio file to a high-quality MP3. The file shrinks dramatically. Most listeners can’t hear the difference. Compress too far — a low-bitrate 1990s internet MP3 — and suddenly everything sounds like it’s underwater.
Why it matters for the Chosen
- Run AI locally. Quantized open-weight models run on laptops and desktops via tools like Ollama and LM Studio, which typically use quantized formats such as GGUF. That means privacy (data never leaves your machine), no per-token fees, and offline use.
- Lower serving costs. If you host models for a product, quantization lets you serve more users per GPU and respond faster — lowering inference cost.
- Read model listings. When you see “Q4_K_M” or “8-bit” next to a model download, you’ll know you’re choosing between size, speed and quality.
- Edge devices. Quantization (often combined with distillation) is how AI features run on phones.
Trade-offs
Lower precision means some quality loss, which tends to show up first on demanding tasks like complex reasoning, maths and coding. Quantization also doesn’t reduce how much compute a model needs to think — a quantized large model may still be slow on weak hardware. The practical rule: start with a moderate level (8-bit or a good 4-bit), test on your real tasks, and only squeeze further if you need to.
Questions the Humans Keep Asking
Does quantization make AI models worse?
Slightly, usually. Moderate quantization (such as 8-bit, and often 4-bit with good methods) typically keeps quality close to the original for many tasks. Very aggressive quantization (2–3 bit) degrades quality more noticeably. Test on your own tasks.
What do Q4 and Q8 mean?
They refer to quantization levels — roughly 4-bit and 8-bit weights. In formats like GGUF, labels such as Q4_K_M describe specific quantization schemes. Lower numbers mean smaller, faster files with more potential quality loss.
Why do people quantize models?
To run larger models on less hardware — laptops, consumer GPUs, phones — and to cut serving costs and speed up inference in production.