This is probably one of the first questions that came to my mind when thinking about artificial intelligence in pharmaceutical manufacturing and quality systems. I first started questioning this during the GMP Pharma Congress, in April 2025. During a presentation about the introduction of AI into pharmaceutical operations, I was surprised by how much the approach resembled the validation of traditional software.

In GMP, we are familiar with this process. We assess the risks, control changes, define what a system should do, validate it, and periodically review its performance. If something changes, we assess the impact and decide whether revalidation is required.It makes sense because, in principle, we understand what the system is designed to do and can identify when something has changed.

• But can we apply exactly the same approach to AI?
• How is AI different from traditional software?

Traditional software follows instructions written by programmers. Machine learning, on the other hand, identifies patterns from data during training and uses them to make predictions or recommendations. AI is even more complicated.

Let’s use an example.

With traditional software, we could establish a rule that a temperature excursion must be escalated to QA. With AI or ML (machine learning), we could instead train a model using thousands of historical deviations and their final classifications. The model might discover that certain temperature excursions, under particular conditions, were rarely escalated.

Nobody necessarily programmed that specific relationship. The model learned it from the data. However, not all models continue learning once deployed. A static model is trained, tested, validated and then frozen. It continues applying the patterns it learned, but does not independently learn from new information.

An adaptive model can continue learning or be updated over time, potentially changing its behavior. This distinction is particularly important for GMP because a model that changes after validation creates a different challenge from one that remains fixed.

The challenge of keeping adaptive AI validated, recognised on the draft of Annex 22 of EudraLex Volume 4 as for now, for critical GMP applications, it focuses on static machine learning models with deterministic outputs, rather than models that continue learning during operation.

In a way, this brings static AI/ML closer to something familiar to the pharmaceutical industry: a defined, controlled and validated system.

Static models are easier to validate because they do not change, but as a paradox, this also means that any weaknesses learned during training may persist throughout their use. Validation should help identify these weaknesses, although it cannot guarantee that all of them will be detected.

On the flip side of the coin, if a model continues learning and its behavior changes after validation, how can we demonstrate that its outputs remain suitable for their intended use? The question is no longer just whether the system performed correctly when it was validated, but whether we can continue demonstrating that it remains suitable and controlled throughout its use.

Also, can the manufacturer properly assess the impact of that update on its validated process? And yet, the GMP responsibility does not disappear simply because the technology is controlled by someone else. The supplier may control the model, but the pharmaceutical manufacturer remains responsible for how it is used and for its impact on product quality and patient safety.

There is also the question of who controls the technology. Many AI/ML systems are developed and maintained by external providers. A pharmaceutical company may validate how the system is used within its own process, but may not have full visibility of the underlying model, its training data or every change introduced by the supplier. Some of this information may also be commercially confidential. Imagine a pharmaceutical company using an external AI system to support deviation investigations. Can the supplier updates the model to improve its performance, but the manufacturer receive limited information about what changed?

Interestingly, these challenges were among the subjects for the EMA Multistakeholder Workshop on Annex 22 in June July 2026. The discussion explored whether adaptive, probabilistic and generative AI could also be accommodated within GMP through appropriate risk-based controls.

This suggests that the regulatory discussion is evolving, although many practical questions remain open. There is another aspect that I find particularly interesting, and it goes beyond validation. Returning to our deviation example, imagine an AI model identifying that 98% of similar historical deviations were classified as minor and recommending the same classification. But does that mean the current deviation has no significant impact on product quality or patient safety? Not necessarily. The model may have learned how deviations were classified in the past, rather than whether those classifications were scientifically justified.

Recognising a pattern is not necessarily the same as understanding why it exists.

This is not, by itself, an ethical problem. We have used statistical tools in pharmaceutical manufacturing for decades, and AI may identify risks that humans would otherwise overlook. The concern is how we start using and trusting these recommendations. If an AI system performs much of a deviation investigation, suggests root causes and proposes CAPAs, the human reviewer may gradually become more of an approver than an investigator. Over time, could this reduce our ability to recognise when the system is wrong?

And if we continue holding people responsible for decisions increasingly shaped by AI, are we giving them enough knowledge, information and control to exercise that responsibility? Perhaps in 2025 it was simply too early for that discussion. But technology was already progressing, and regulation does not always move at the same speed.

In pharmaceutical manufacturing, there are very good reasons to prefer AI/LM systems, but I believe we need to distinguish between what AI can do, what we allow it to do and how much we understand about the decisions it supports. The challenges that remain are both technical and ethical. We need to ensure that users knows sufficiently  the model than the company  is using. At the same time, we should be careful not to gradually replace human judgement with automated recommendations while continuing to assign humans full responsibility.

If AI or ML increasingly influences decisions affecting medicine quality and patient safety, how do we ensure that human oversight remains meaningful, rather than becoming just a signature at the end of the process?

---

Regulatory context

For the current EU GMP framework, see EudraLex Volume 4, including the applicable Annex 11. The European Commission consultation on revisions to Annex 11 and proposed Annex 22 is a separate, draft regulatory development.