Small models, distillation and the case against always using the biggest

The early scaling story was that bigger models were better, and it was true. What has happened since is that small models have improved faster than large ones, for reasons worth understanding.

Why small models improved so much

Better data. The largest single factor.

Early large models were trained on relatively unfiltered web scrapes. Careful filtering, deduplication and curation produce substantially better models at the same size.

Synthetic data — generated by a larger model and filtered for quality — has extended this further, particularly for reasoning and instruction-following.

Longer training. Work on the relationship between model size and training tokens showed that many early models were undertrained relative to their parameter count.

Training a smaller model on far more data produces a model that is cheaper to serve and performs comparably, which is exactly the trade a deployer wants.

Distillation. Training a small model to reproduce the outputs of a large one.

The small model learns from the large model's full output distribution rather than only from hard labels, which transfers more information than training on the original data alone.

Architectural improvements, including attention variants that reduce memory requirements and mixture-of-experts designs that activate only part of the network per token.

That last one is worth noting: a mixture-of-experts model may have a very large total parameter count while activating a small fraction per token, which decouples capacity from inference cost.

Why it matters commercially

Inference cost scales with model size, and inference happens constantly.

A model an order of magnitude smaller is roughly an order of magnitude cheaper to serve, faster to respond, and can run on hardware that the large one cannot.

For a service handling large volumes of similar requests, the difference between a frontier model and a well-chosen small one is the difference between a viable business and an unviable one.

On-device inference

The application that only small models enable.

Running locally means no network round trip, no per-request cost, availability offline, and — most significantly — data never leaving the device.

That last point is the compelling one for anything involving personal or confidential information.

The constraints are memory, which bounds model size, and power, which bounds sustained throughput on battery.

Quantisation is essential here, and models in the low billions of parameters at four-bit precision now fit comfortably on current phones and laptops.

Choosing a size

The practical method, which almost nobody follows.

Build an evaluation set from your actual task. Start with the smallest plausible model. Measure. Escalate only if it fails.

The common alternative — start with the largest available and never revisit — leaves a great deal of money on the table and generally adds latency users notice.

Many production tasks are classification, extraction, formatting or summarisation, and small models handle all of them well.

Routing

The architecture that captures most of the benefit without committing to one model.

A classifier examines the request and directs it to an appropriate model: small for routine, large for hard.

Implementation ranges from simple heuristics to a trained router to a cascade, where the small model attempts first and escalates when its confidence is low.

Cascades are attractive because they degrade gracefully and because the escalation rate is measurable, which makes the cost saving auditable.

Fine-tuning a small model

Frequently better than prompting a large one.

For a narrow, repetitive task with plenty of examples, fine-tuning a small model on those examples generally beats a large general model with a long prompt — on accuracy, cost and latency simultaneously.

Efficient fine-tuning methods that adjust a small number of additional parameters make this cheap enough to be routine.

The cost is maintenance: a fine-tuned model must be retrained when the task changes, whereas a prompt can be edited.

Where large models remain necessary

Tasks requiring broad world knowledge across unpredictable domains.

Multi-step reasoning where small models fail in ways fine-tuning does not fix.

Open-ended generation where quality is judged holistically.

Anything genuinely novel, where there is no training set to fine-tune on.

Which is a real and much narrower set of applications than current deployment patterns suggest.