AI & Compute
Inference Costs And Why Serving Is Not Training
Training a model is a single large bill, but serving it is a permanent one, and the two workloads stress completely different parts of a data centre.
Training a model is often described as the expensive part, and the headline figures encourage that reading. Serving the finished model to users is the cost that never stops, and it behaves quite differently.
Training is a burst and serving is a baseline
A training run occupies a large cluster continuously for weeks or months, then ends. The spending is concentrated, planned in advance, and can be scheduled around hardware availability.
Serving is the opposite shape. Every request costs a small amount of compute, and the total is set by how many people use the product rather than by any decision the developer makes once.
That difference matters for anyone assessing whether an AI product can work economically. A model that is cheap to train and heavily used may still lose money on every interaction.
The hardware is used in a different way
Training runs push chips to sustained high utilisation, with large batches of data flowing through in parallel. The workload is predictable, and the cluster can be kept saturated almost all the time.
Inference arrives in unpredictable bursts. Requests come when users are awake, sizes vary, and the system must hold spare capacity for peaks it cannot schedule.
Idle accelerators still cost money, so serving economics depend heavily on how well traffic can be batched together without making any individual response feel slow.
Memory bandwidth usually binds before raw speed
Generating text means reading the model's parameters repeatedly, once for each token produced. The chip spends much of its time moving numbers rather than multiplying them.
This makes memory bandwidth, not arithmetic throughput, the limit that most often decides how fast a model responds. A faster processor with the same memory system may deliver little improvement.
It also explains why smaller models are disproportionately cheaper to serve. Fewer parameters means less data moved per token, which compounds across every request the system handles.
Response length is a cost multiplier
A long answer costs more than a short one because each additional token requires another pass through the model. Output length is a direct lever on serving cost.
Input length matters too, though differently. A long prompt is processed in one go, but it must then be carried through the generation of every subsequent token.
Products that quietly attach large amounts of context to each request can therefore be far more expensive to run than their interface suggests.
Optimisation happens after the model is finished
Most of the work that makes serving affordable occurs once training is complete. Techniques that reduce numerical precision, share computation between requests or cache repeated work all cut cost without retraining.
These changes carry trade-offs in accuracy and consistency, and the acceptable amount varies by application. A coding assistant and a search summary tolerate different kinds of degradation.
The practical consequence is that two deployments of the same underlying model can differ substantially in both cost and behaviour, depending on choices made entirely at serving time.
