From draft Annex 22 to the latest regulatory discussions, what might inspectors expect us to demonstrate about AI?
Over the last few months, I have been following the discussions around the introduction of Artificial Intelligence into pharmaceutical manufacturing and quality systems.
What I found interesting was how the discussion seems to be moving from which AI technologies we should allow towards something more practical: can we actually demonstrate that we understand and control the AI we are using? Perhaps I have been influenced by this in my previous posts, or perhaps it is such an obvious question that we cannot stop asking ourselves. Because, eventually, inspectors will be asking us the same thing.
But this time, let’s change the question slightly. Do we actually know where AI is being used?
At first, it sounds almost obvious. But AI is increasingly embedded within software, supplier platforms and applications that we may already use every day. A system may introduce new AI functionality through an update, sometimes without users fully appreciating how it works.
The original draft Annex 22 already emphasised the importance of defining the intended use, understanding the model, identifying risks and maintaining appropriate documentation. The subsequent EMA workshop explored whether these principles could also support more complex technologies, including adaptive and generative AI, provided suitable controls can be demonstrated. More complex technologies to understand, and potentially more difficult to control. Not to mention that, as of the publication date of this post, Annex 22 is still a draft. Technically, we need to comply with the current Annex 11 and the other applicable GMP requirements.
So, before discussing validation or technical safeguards, organisations need to understand which AI applications they are actually using. Imagine a Quality department using an AI-enabled system to support deviation investigations. The company may have qualified the supplier and validated the software, but does it understand which parts of the investigation are being influenced by AI? What data is being processed? Can the underlying model change without the company being aware?
These seem like reasonable questions for an inspector to ask.
And they bring us back to something familiar in GMP: we cannot control what we do not know is there. Can we trust the output, or are we simply trusting the system?
Another major theme was the reliability of AI-generated information.
The discussion recognised that the controls we can put in place may have limitations. They may detect known errors but fail to identify unexpected ones. During the training, the practical message was that users should verify AI outputs against the original information and understand the limitations of the technology. But that is the whole point. How many people in Quality are able to explain a technology that is evolving faster than most of us can understand? And when I say most of us, I include the IT geeks.
Imagine an AI system preparing the first draft of a deviation investigation. It reviews hundreds of historical records and suggests a possible root cause. The investigator reads the explanation. It looks logical, well structured and consistent with previous cases. But what if the historical investigations contained weaknesses that are now hidden behind the polished explanation provided by AI? What if the AI overlooked an important piece of evidence?
And, perhaps more importantly, what if the investigator becomes so accustomed to receiving apparently reasonable recommendations that they gradually stop questioning them? This is where human oversight becomes particularly interesting. Both draft Annex 22 and the EMA discussions recognise the importance of qualified personnel and appropriate human involvement.
Qualified personnel. Not exactly the most specific term, although in this case I can understand why. The challenge may be ensuring that people continue to exercise independent judgement when technology increasingly performs the initial analysis for them.
Are we preparing people as well as validating systems?
This was another interesting point from the training. We spend considerable effort qualifying equipment, validating computerised systems and maintaining them under change control. But what about the people using AI? The training highlighted AI literacy, practical experience, critical thinking and the ability to question outputs. This aligns with the broader regulatory discussion around competence, accountability and meaningful human oversight.
For example, someone using an LLM to review a Product Quality Review may need to understand more than how to write an effective prompt. They should recognise when information is missing, distinguish evidence from an AI-generated interpretation and know when a recommendation requires further investigation.
Of course, this does not mean everyone needs to become a data scientist. But perhaps we need to reconsider what being adequately trained means when a system can produce convincing answers that are not necessarily correct. And I think there is an ethical dimension here.
If we give people responsibility for reviewing AI decisions, do we also give them the knowledge, time and authority to challenge those decisions?
Can an employee at a small site within a global organisation realistically challenge a decision made by an AI system rolled out across the entire company?
Such a challenge might require a global investigation, changes affecting multiple affiliates, updates to the system and potentially revalidation. And what happens between discovering an incorrect decision and the time needed to investigate, understand and correct the problem? What happens to everyone else using that same system during this period? Otherwise, we risk maintaining human accountability on paper while gradually reducing meaningful human involvement.
What happens after AI has been implemented?
Another common theme across the regulatory documents and the training was the need for ongoing control.The training translated this into a more practical question: how do we know when an AI system has changed, and what do we do about it?
This is particularly relevant when AI functionality is provided by an external supplier.
A company may validate a system today, but what happens when the supplier updates the model next month? How do we assess whether its performance remains acceptable?
A periodic review may help, but its effectiveness depends on what information the company can access and whether important changes can be detected in time. Again, none of this is entirely new to GMP. What may be different is the complexity of the technology and how much visibility we have over its operation.
And where does responsibility remain?
One message seems consistent throughout these discussions: introducing AI does not remove the responsibility of the pharmaceutical company or the professionals making GMP decisions. But I think there is a difference between assigning responsibility and making sure that someone can genuinely exercise it.
A Quality Manager or QP can be held accountable for decisions supported by AI. Yet accountability becomes more difficult if the organisation cannot adequately explain the system’s behaviour, verify its conclusions or identify when something has gone wrong.
A company could have an AI policy, training records, a risk assessment and a validation report, yet still fail to recognise when its people are relying too heavily on the system.
Perhaps the real challenge is not simply demonstrating that AI has been validated, but demonstrating that the organisation remains capable of understanding, questioning and controlling what AI does throughout its use.
And this brings me to the question I would leave open.
As AI becomes more capable of supporting our GMP decisions, how do we make sure that our ability to challenge those decisions does not become weaker?
And could we ever rely on AI to validate itself?
References
- Draft Annex 22 — Artificial Intelligence, EudraLex Volume 4 (2025 consultation)
- EMA Multistakeholder Workshop — Annex 22, report published 2 October 2026
Leave a Reply