What Is Quantization in AI? A Beginner-Friendly Explanation
Quantization is a model compression technique that stores a model's numbers (its weights and activations) using fewer bits, so the model takes up less memory and runs faster, while keeping most of its accuracy. In practice it often means moving from 16- or 32-bit floating-point numbers down to 8-bit or 4-bit integers.
The everyday analogy
Think of a high-resolution photo saved as a large file. You can shrink it to a quarter of the size, and on a phone screen it still looks almost the same. The fine detail you removed is not noticeable in normal use.
Quantization does this with an AI model. A large model is really just billions of learned numbers. If you round those numbers onto a coarser grid, the model becomes much smaller, yet its behavior stays close to the original because neural networks are surprisingly tolerant of rounded values.
Bits and precision: what the numbers mean
A model trained in the usual way keeps its weights as 32-bit floating-point numbers, written FP32. Each one can hold a lot of fine detail. Lower formats use fewer bits:
| Format | What it is | Typical role |
|---|---|---|
| FP32 | 32-bit floating point | Full training precision |
| FP16 / BF16 | 16-bit floating point | Common serving format |
| INT8 | 8-bit integer | Widely used, near-full quality |
| INT4 | 4-bit integer | Smallest, small quality trade-off |
Each step down roughly halves the memory needed per value. The trade is deliberate: you give up a small, bounded amount of numerical detail in exchange for large gains in size, cost, and latency.
Why it matters for large models
Modern models are big. A model with tens of billions of parameters can need more memory than a single accelerator provides, which makes it slow or expensive to run. Without quantization, teams pay for more memory and tolerate longer response times.
NVIDIA's technical documentation gives a concrete example: a Llama 2 7B model stored in FP16 needs about 14 GB of memory, and moving it to FP8 cuts that to roughly 7 GB in one step. The same pattern appears across model families, which is why quantization is a standard step before deploying on phones, laptops, and edge devices.
How the conversion works
At a high level, quantization maps a range of high-precision numbers onto a smaller set of integer levels. Two values are used to describe the mapping: a scale that sets the spacing between levels, and a zero-point that anchors where zero falls. At inference, the hardware does fast integer arithmetic, and results are converted back when needed.
Two broad approaches decide when this happens:
- Post-training quantization (PTQ) applies the conversion to a model that has already finished training. It is quick and needs little or no retraining, sometimes with a small calibration set to keep accuracy steady.
- Quantization-aware training (QAT) simulates low-precision arithmetic during training itself, so the model learns to work with rounded values. It often preserves slightly more accuracy but costs more effort.
For very large language models, named methods such as GPTQ and AWQ handle the conversion in ways that protect the most important weights.
What you actually gain and lose
| Dimension | Effect of quantization |
|---|---|
| Model size | Smaller, sometimes several times over |
| Memory use | Lower, including during inference |
| Speed | Faster on hardware built for low-precision math |
| Energy and cost | Reduced |
| Accuracy | Usually a small drop, growing at aggressive levels |
INT8 is often close to lossless for everyday tasks. INT4 brings a measurable but usually modest change, and pushing below that can degrade quality quickly. The honest way to choose a level is to measure the quantized model against the original on your own tasks, rather than assume a single setting fits everything.
How it differs from other compression methods
Quantization is one of several ways to make a model lighter. Pruning removes weights that contribute little, and knowledge distillation trains a smaller "student" model to mimic a larger "teacher." Quantization keeps the same model structure but represents it with simpler numbers, so it is often the easiest first step and is combined with the others.
If you want to try this on your own computer, the guide to running local AI models explains how tools use quantized files to fit capable models on ordinary hardware.
FAQ
Does quantization change what the model knows?
It changes how the numbers are stored, not the model's overall design. The model keeps the same skills, with a small accuracy trade-off that depends on the precision chosen.
Is quantization the same as compression like a zip file?
No. A zip is lossless and restores the exact original. Quantization genuinely rounds the values, so some detail is discarded, which is why accuracy has to be checked.
Will a quantized model run on my laptop or phone?
Often, yes. Lower memory use is the main reason large models can run on consumer devices, though speed still depends on the hardware and model size.
What is the difference between PTQ and QAT?
PTQ converts an already-trained model and is fast to apply. QAT trains with low precision simulated and usually retains a little more accuracy at the cost of extra work.
Does a lower bit number always mean worse results?
Lower bits add more risk, but a well-chosen 4-bit or 8-bit model can be very close to the original. The difference should be measured on real tasks rather than assumed.
Key takeaways
- Quantization stores model numbers with fewer bits, from FP32 down to INT8 or INT4.
- It cuts memory, size, energy, and latency while usually keeping most accuracy.
- PTQ is the quick route; QAT preserves more accuracy with extra effort.
- The right precision is a measured trade-off, not a one-size-fits-all choice.
Continue with how to run local AI models and what is edge AI.
Sources
Learn by doing
Read a guide, then practice with bite-sized AI lessons you can take offline.
Start learning free