How models are actually deployed

Training a model produces a set of weights. Turning that into a service that answers millions of requests reliably and affordably is a separate discipline.

The two phases of generation

Understanding serving requires understanding that generation has two distinct phases with different bottlenecks.

Prefill. Processing the input prompt. All input tokens can be handled in parallel, so this phase is compute-bound and uses the accelerator's arithmetic capacity efficiently.

Decode. Generating output tokens one at a time, each depending on the last. This is inherently sequential and memory-bandwidth-bound: each token requires reading the model weights, and the arithmetic units sit largely idle.

The practical consequence is that a long prompt with a short answer and a short prompt with a long answer have very different cost profiles, which is why providers price input and output tokens differently.

Batching

The single largest efficiency lever.

Because decode is memory-bound, processing several requests simultaneously amortises each weight read across all of them at almost no extra time cost.

Naive batching waits for a batch to fill and finishes when the slowest request completes, wasting capacity.

Continuous batching adds and removes requests from the running batch as they arrive and complete, keeping the accelerator busy. This is now standard and delivers large throughput gains.

It also creates a genuine tension: larger batches improve throughput and increase latency for individual requests, which is why services offer different tiers.

The key-value cache

The memory structure that dominates serving capacity.

During generation, the model caches intermediate values for every previous token so it does not recompute them. That cache grows with sequence length and with batch size.

For long contexts and large batches, the cache can consume more memory than the model weights.

Which makes cache management the central engineering problem in serving. Techniques include paged allocation — borrowing ideas from operating system virtual memory to avoid fragmentation — and sharing cache between requests with common prefixes.

That last point matters commercially: prompt caching, where a long shared system prompt is computed once and reused, is a large cost saving and is now offered explicitly by several providers.

Quantisation

Storing weights at reduced precision.

Moving from sixteen-bit to eight-bit roughly halves memory footprint and bandwidth requirements, with accuracy loss that is small for most tasks when done carefully.

Four-bit and lower are used in practice, with more noticeable degradation that varies by model and task.

The important point is that quantisation improves throughput mainly by reducing memory traffic, not by reducing arithmetic — which follows directly from decode being memory-bound.

Routing

An increasingly standard architecture.

Most production traffic does not need the largest available model. Simple queries can be served by a small fast model at a fraction of the cost.

A router classifies incoming requests and directs them accordingly, escalating only where needed.

Done well this cuts serving cost substantially with little quality impact. Done badly it produces inconsistent behaviour that users notice.

Speculative decoding

A technique that exploits the memory-bound nature of decode.

A small fast model proposes several tokens ahead; the large model verifies them in a single parallel pass, accepting the prefix that matches what it would have produced.

Because verification is parallel and therefore cheap in bandwidth terms, this produces a genuine speedup with mathematically identical output.

The operational realities

Cold starts. Loading a large model into accelerator memory takes time. Services keep models resident, which means capacity is provisioned rather than elastic.

Tail latency. Average latency is easy; the ninety-ninth percentile is what users experience as slow. Long generations and queueing dominate the tail.

Capacity planning. Demand is spiky, accelerators are scarce and expensive, and overprovisioning is costly while underprovisioning produces rate limits.

Evaluation in production. Model updates change behaviour in ways benchmarks do not capture. Serious deployments run shadow traffic and staged rollouts.

What this means for anyone building on top

Cost is driven by tokens, so shortening prompts and constraining output length are the direct levers.

Structure prompts so the stable part comes first, to benefit from prefix caching.

Use the smallest model that passes your own evaluation, not the largest available.

And measure latency at the tail rather than the mean, because that is the number your users will describe.