AI & Compute
Quantization And What Precision You Can Afford To Lose
Storing model weights with fewer bits shrinks memory and speeds serving, and the practical question is which parts of a network tolerate the reduction without visible degradation.
Models are trained using relatively precise numbers and are frequently served using much coarser ones. Understanding why that works, and where it stops working, explains a large share of current deployment engineering.
Precision costs memory and bandwidth
Every weight occupies space, and serving requires reading all of them for each step of generation. Halving the number of bits halves both the memory needed and the data moved.
Since most inference is limited by memory bandwidth rather than arithmetic, that reduction translates almost directly into speed.
It also determines whether a model fits on available hardware at all, which is often the difference between deployable and not.
Networks tolerate approximation unevenly
Neural networks are robust to small perturbations in most of their parameters, because the output depends on the aggregate behavior of many values rather than on any single one.
Some components are exceptions. Certain layers and a small number of unusually large activation values carry disproportionate influence, and coarsening them degrades output noticeably.
Practical schemes therefore reduce precision aggressively in the bulk of the network while keeping sensitive parts at higher resolution.
Calibration decides how the range is mapped
Reducing bits means choosing which range of values the available levels represent. A range set too wide wastes resolution; too narrow and extreme values are clipped.
Calibration runs sample data through the model to observe actual distributions, then sets scaling per layer or per group of weights rather than globally.
Finer grouping preserves accuracy and adds bookkeeping, which is the central trade in the design of a quantization scheme.
Training with reduction in mind works better than converting afterward
Converting a finished model is fast and requires no retraining, which suits most deployments. Where quality loss matters, the model can instead be trained or fine-tuned while simulating the reduced precision.
The network then adapts its parameters to the constraint, and the resulting model holds up at lower precision than post-conversion methods reach.
The cost is access to training infrastructure and data, which is why the simpler approach dominates in practice.
Measuring the loss is the hard part
Aggregate benchmark scores can remain almost unchanged while specific capabilities degrade, particularly long reasoning chains, rare languages and precise formatting.
Teams that evaluate only on a general benchmark frequently ship a reduction that users experience as the model becoming subtly worse at the thing they use it for.
Comparing outputs on the actual workload, before and after, is the only reliable check available.
