Where AI systems actually fail in production

A model that performs well in evaluation and poorly in production is the normal case rather than the exception. The reasons are consistent across domains.

Distribution shift

The single largest cause.

A model learns the relationship present in its training data. When the world changes, that relationship stops holding.

Covariate shift — the inputs change. A vision system trained on images from one camera degrades when the camera is replaced.

Label shift — the distribution of outcomes changes. A fraud model trained on last year's fraud patterns misses this year's.

Concept drift — the relationship itself changes. What predicted churn before a pricing change does not predict it after.

The critical property is that a model does not know it has drifted. It produces confident outputs on inputs unlike anything it was trained on.

Which is why monitoring input distributions, not just output accuracy, is the practical defence. Accuracy is frequently unavailable in production because labels arrive late or never.

Training-serving skew

The most common engineering bug in machine learning systems.

Features computed one way during training and another way at serving time. Different code paths, different libraries, different handling of missing values, different time zones, different rounding.

The model receives inputs subtly unlike those it was trained on and performance degrades in ways that are hard to trace.

The structural fix is to compute features through the same code in both paths — the motivation behind feature stores — rather than to be careful.

Leakage

The reason a model looked excellent in development and fails in production.

Information available at training time that will not be available at prediction time, or that encodes the answer.

Classic cases: a field populated only after the outcome occurs; an identifier correlated with the label; a timestamp that reveals which cases were reviewed; normalising across the whole dataset before splitting.

Leakage produces impressive validation results and no real capability, and it is discovered after deployment.

The defence is temporal splitting — train on the past, validate on the future — which mirrors how the model will actually be used.

Feedback loops

The failure mode with the worst long-term consequences.

A model's predictions influence the world, which generates the data the next model is trained on.

A recommendation system shows items it predicts will be clicked. Those items get clicks. The next model learns they are popular. The catalogue narrows.

A resource allocation model directs attention to areas it predicts will need it. Those areas generate more recorded incidents. The model's belief is confirmed by data it caused.

This produces systems that appear to perform well by every available metric while progressively diverging from reality, and it is genuinely difficult to detect without deliberate randomisation — holding out a fraction of decisions to gather unbiased data.

Edge cases and the long tail

Aggregate accuracy conceals distribution.

A system performing well on average may fail systematically on a subgroup that is small in the data and important in practice.

Evaluation stratified by relevant subgroups, rather than reported as a single figure, is the minimum. This is how most documented fairness failures were eventually found.

Operational failures

Unglamorous and frequent.

An upstream data source changes schema without notice. A pipeline fails silently and serves stale features. A dependency updates and changes numerical behaviour. Latency rises under load and requests time out.

These are ordinary software reliability problems, and machine learning systems are frequently built with less engineering rigour than the surrounding services.

The human element

Automation bias. Operators over-trust a system that is usually right, and stop checking.

Alert fatigue. A system producing too many low-value flags gets ignored, including the valuable ones.

Workflow mismatch. A prediction delivered at the wrong moment, or in the wrong place, is not acted on regardless of accuracy.

These determine realised value more than model quality does, and they are usually discovered after deployment because they are not tested for.

What a serious deployment includes

Monitoring of input distributions, not only outputs.

A held-out randomised fraction to gather unbiased evaluation data.

Stratified performance reporting by subgroup.

Shadow deployment before switching traffic, and staged rollout after.

A defined retraining trigger and process, rather than retraining when someone remembers.

And a documented rollback path, because the most useful property of any production system is the ability to turn it off quickly.