AI & Compute
Why Synthetic Training Data Is Useful And Risky
Generating training data with models solves scarcity and privacy problems, but it also risks narrowing diversity and quietly amplifying whatever flaws the generator already has.
Models are increasingly trained on data produced by other models. The practice addresses real shortages, and it introduces failure modes that are difficult to detect from the outside.
Real data is scarce in specific ways
Broad text and images are abundant, but data covering unusual situations is not, and those are often exactly the cases a system must handle correctly.
Rare events are rare by definition, so collecting enough natural examples of them can take years or may not be possible at all.
Generating those situations deliberately lets a training set be balanced in a way that observation alone cannot achieve.
Privacy and licensing push in the same direction
Data describing people carries legal and ethical constraints that limit how it may be stored, shared and reused across projects.
Synthetic substitutes that preserve statistical structure without corresponding to any real individual can relieve some of that burden, though the protection is not automatic.
A generator trained on sensitive records can reproduce fragments of them, so a synthetic label does not by itself guarantee that nothing identifiable remains.
Diversity narrows without anyone noticing
A generator produces samples reflecting the distribution it learned, and it tends to produce typical examples more often than unusual ones.
Training on that output shifts the next model further towards the centre of the distribution, and repeating the cycle compounds the effect.
The consequence is a gradual loss of the tails, which is precisely where the difficult and interesting cases live.
Errors propagate as though they were evidence
Whatever the generating model gets wrong appears in the synthetic data as confident, well-formed content indistinguishable in style from correct material.
A downstream model has no way to identify those errors and will learn them alongside everything else, sometimes more firmly because they appear consistently.
Filtering with another model risks the same problem, since the filter shares the biases of the generator when both derive from similar sources.
Verification is what makes the approach work
Synthetic data is most reliable in domains where an independent check exists, such as code that can be executed or mathematics that can be verified.
In those settings, generated candidates can be tested rather than trusted, and only the ones that pass enter the training set.
Where no such check exists, the safest practice is to mix synthetic material with real data and to monitor for the narrowing that otherwise accumulates unseen.
