In July 2024, researchers from Oxford, Cambridge and other universities published a study in Nature titled “AI models collapse when trained on recursively generated data“. Their findings raise a concern that could become particularly relevant for the pharmaceutical industry as AI becomes increasingly involved in GMP processes. The researchers found that successive generations of AI models trained recursively on data generated by previous models can gradually lose information from the tails of the original data distribution. Rare events, unusual patterns and less frequently represented information begin to disappear. Over generations, the models may become less representative of the original information, while errors and distortions accumulate.

Imagine making a photocopy of a photocopy, and then another photocopy of that copy. Each generation may lose some detail and resolution perhaps even substantially different from the original.
In the case of language models, this raises the possibility of progressively less diverse outputs, favouring common patterns while losing less frequent ones. Under certain conditions, this deterioration leads to what researchers call “Model Collapse”.

Maybe a long shot, but let’s imagine what could this mean for GMP. Imagine an AI system supporting pharmaceutical deviation investigations. Initially, it learns from historical records prepared by experienced quality professionals (our legacy), including unusual failures, uncommon root causes and investigations requiring considerable judgement. The beauty of a human mind can be that is unpredictable and for its variety of outcomes, like colours with different, infinite shades and pattern.

Let’s move the clock forward very quickly. AI contributes to new investigations. Its recommendations are reviewed, incorporated into reports. Aapproved. These records eventually become part of the training material for future AI models.

Could successive generations progressively lose the less common patterns present in the original investigations? Will all the colours become either red, blu or green? Or just black and white.

The possibility deserves attention. Some of the most important lessons in pharmaceutical manufacturing come from unusual events that may have occurred only once. And also the biggest unexpected discoveries..
A model might become increasingly consistent in handling routine investigations while losing the ability to recognise less familiar situations. Its overall performance could remain apparently satisfactory, even as its coverage of rare but important cases deteriorates.

This is a hypothetical application of model collapse. But then.. what if..

What if humans begin losing resolution too? There is another side to this problem, one that concerns us rather than the machines. Consider something simple: 9 × 7. Yes is an old story ( Isaac Asamov Feeling of Power, published in 1956 and summarize at the end of the post for the most curious readers).

Many of us can calculate it immediately. But imagine spending years using a calculator for every elementary operation. We might eventually find ourselves reaching for the calculator even when the answer should be familiar. This does not necessarily mean that we have permanently lost the ability to calculate. Perhaps we have simply become less accustomed to exercising it.But what happens when the capability we stop exercising is not mental arithmetic, but critical thinking? Much easier to loose.

We increasingly use AI to summarise information, develop arguments, compare alternatives and suggest conclusions. These are useful applications, but they can also reduce the amount of independent reasoning required from us. A 2025 study by Microsoft Research and Carnegie Mellon University found that greater confidence in generative AI was associated with less reported critical thinking during AI-assisted tasks (it can be discussed for its limitation, but let’s just assume it as a possibility). It did not establish that people permanently lose their reasoning abilities, but it raises a relevant concern. What if, over time, we become less accustomed to questioning assumptions, identifying contradictions or developing our own interpretations? But yet responsible of the outcome. But also what if this is the safest option at the beginning, until the model collapse kicks in.. Lot of what if, I know.

The information we produce might become more polished and consistent, yet potentially less diverse. If humans increasingly reproduce AI-generated interpretations, and future AI systems learn from those same interpretations, could both processes reinforce one another? AI could gradually lose less common information through recursive training, while humans produce fewer independent perspectives capable of replenishing that information.

An alternative ending to Asimov’s vision

This is a speculative scenario. It is also true that a sufficiently advanced AI could instead develop effective ways to verify information independently, generate new knowledge and prevent such degradation. There is no evidence that the singularity would inevitably lead to collapse. As AI contributes to quality records, investigations and professional decisions, preserving original evidence may become increasingly important. So might preserving the ability of the professionals reviewing those outputs to reason independently. Perhaps the longer-term concern is not simply that AI could forget information. It is that we might gradually stop exercising some of the abilities needed to recognise what AI has forgotten.