AI & Compute
Evaluating an AI product claim
A short checklist for reading announcements, which mostly consists of asking what was measured and against what.
Most AI product claims are not false. They are true about a narrow thing and presented as true about a broad one. A few questions separate the two.
What exactly was measured
The first and most productive question.
"Outperforms human experts" — at which task, with which experts, under what time pressure, with what resources available to them?
Comparisons frequently pit a model with unlimited time against professionals working under realistic constraints, or against a small and unrepresentative expert sample.
"Ninety-five percent accurate" — on what distribution? Accuracy on a balanced test set and accuracy in production where one class dominates are entirely different numbers.
For rare events, a system that always predicts the common case can be highly accurate and useless. Precision and recall, or the full confusion matrix, are the informative reporting.
What is the baseline
Frequently absent, and its absence is informative.
A model that achieves a given score should be compared against the simplest reasonable alternative: a rule-based system, a small classical model, or the existing process.
A surprising proportion of machine learning deployments fail to beat a well-tuned simple baseline, and the comparison is rarely published.
Was it evaluated on data it could have seen
The contamination question.
For anything using a model trained on web data, any public benchmark or dataset may be in the training corpus.
Ask when the evaluation data was created relative to the training cutoff, and whether held-out or freshly generated items were used.
Who ran the evaluation
Vendor-run evaluations of vendor products are not worthless and should be weighted accordingly.
Independent evaluation, replication by third parties, and published methodology that allows others to check are the standards that apply in any other technical field.
What happens when it fails
Usually the most important question and almost never in the announcement.
Every system fails. What matters operationally is the failure mode.
Does it fail visibly or silently? Does it fail safely or produce a confident wrong answer? Can a user tell? Is there a confidence signal, and is it calibrated?
A system with a modest accuracy that reliably flags its own uncertainty is more useful than a more accurate one that is uniformly confident.
What is the cost per unit of work
Increasingly relevant as approaches that spend more compute per query become common.
A system that achieves better accuracy by sampling many candidate answers and selecting among them is trading cost for quality. That is a legitimate trade and it should be stated.
Comparing accuracy without comparing cost and latency omits half the decision.
Is the demonstration representative
For anything shown as a video or a live demo.
How many attempts. Whether the input was chosen or arbitrary. Whether the environment was prepared. Whether a human was in the loop. Whether it is real time.
None of these questions is hostile. Good demonstrations answer them unprompted.
What data was used, and whose
Relevant for both legal and quality reasons.
Training data provenance affects licensing exposure. It also affects performance: a model trained on data unrepresentative of your population will underperform on it, and that gap will not appear in the vendor's evaluation.
This is the mechanism behind most documented cases of systems performing worse for particular groups — the training distribution did not match the deployment distribution.
What is the actual product
A distinction worth making explicitly.
Many products are a general-purpose model with a prompt and an interface. That can be genuinely valuable — interface and workflow integration are real work — and it is a different proposition from a purpose-built system.
Ask what would happen if the underlying model were unavailable, because that reveals where the value sits.
The procurement version
If you are buying rather than reading, one step outweighs all of the above.
Build an evaluation set from your own data, with your own correct answers, before speaking to any vendor. Run every candidate on it.
A few hundred representative examples scored by someone who knows the domain will tell you more than every benchmark, case study and demonstration combined.
