AI is moving quickly into pharmaceutical manufacturing. But is our ability to understand and control the risks keeping pace?
We are building a new motorway. Until now, we had a reasonable understanding of what vehicles would use it, so we installed barriers, road signs and safety systems to prevent accidents. But are we now entering a new chapter? Can we still say we know what vehicles will travel on it? How heavy they will be, how fast they will move, or how much traffic we should expect? Would we be confident that the barriers we have installed are sufficient?
This is increasingly how I see the introduction of artificial intelligence into pharmaceutical manufacturing. AI is already entering different areas of our industry, from manufacturing processes and quality control, from APIs to finished products, from clinical trials to licensed medicines. But how many people with a quality background really understand how these systems work?
Even those responsible for implementing them may understand how to integrate and operate the technology without necessarily understanding all the complexities of the models behind it. And perhaps this is where the first problem begins.
The road is changing while we are driving on it
We are not just talking about a road that is changing. The vehicles are getting bigger and heavier, traffic is increasing, and everything seems to be moving faster. Could the consequences of an accident become more serious? Our regulatory framework and many of our established GMP controls have evolved following previous failures. We have learned important lessons from events that exposed weaknesses we had not sufficiently recognised, or risks that were recognised but not adequately addressed.
In some cases, we relied too much on people doing the right thing rather than building sufficient safeguards into the process. With AI, we are introducing technologies whose capabilities, applications and limitations are still evolving. And to some extent, we have to rely on systems that we may not fully understand.
Yes, we are building guardrails, but we may not yet fully understand the traffic they will need to contain. And there is another complication.
This is not happening on one road.
AI is being introduced across manufacturing sites, laboratories, suppliers, logistics and quality systems, potentially involving thousands of people with different levels of understanding, different approaches to risk and different degrees of oversight.
Each organisation may assess its own section of the road.
But who is looking at the entire network?
Do we have enough auditors with both AI expertise and GMP knowledge? Or should we be developing this expertise more systematically?
In a typical GMP inspection, computerised system validation is only one of many areas to be assessed. As AI becomes more integrated into critical processes, will this need to take a more central role?
What if the consequences of a mistake are changing?
Think about a car accident. At 30 km/h, the consequences might be relatively limited. At 180 km/h, the same mistake could have a completely different outcome. AI seems to be accelerating things. A mistake in a traditional process might affect a limited number of records or decisions before being identified. An AI system, however, could potentially repeat the same mistake across hundreds or thousands of records, processes or decisions before anyone notices. And if several organisations rely on the same technology, the consequences might extend beyond a single company. Of course, this is not inevitable. AI could also identify problems earlier, improve consistency and prevent errors that humans might overlook. But the question remains whether our ability to detect and contain a mistake is improving as quickly as our ability to reproduce it.
The responsibility of knowing what we do not know
The ongoing discussion around the proposed GMP Annex 22 recognises some of these difficulties, including validation, human oversight, reliability and the use of guardrails. But regulations alone cannot resolve every uncertainty, particularly when the technology continues to develop. We cannot expect everyone working in pharmaceutical manufacturing to become an AI expert. Equally, we cannot simply assume that a system is sufficiently understood because it has been supplied by a technology company, validated against selected criteria or successfully used elsewhere.
There is a difference between knowing how to operate a system and understanding the risks of relying on it. And perhaps there is an even more difficult question about responsibility. I don’t think the answer is to stop innovation until we understand everything. That would probably be unrealistic and potentially counterproductive. But neither should the speed of technological development determine the level of risk we are willing to accept. In GMP, we aim to be proactive, while recognising that unexpected failures can still happen. Perhaps with AI, we should give more attention not only to preventing mistakes, but also to recognising them quickly, containing their consequences and maintaining the ability to intervene when something goes wrong. Because the difficulty may not simply be that we are building a new road. It is that the road, the traffic and the vehicles are all changing at the same time, while we are already driving on it. Are we prepared to accept uncertainty when introducing a technology that could influence product quality and, ultimately, patient safety? The instinctive answer might be that we should not accept uncertainty. But can we really eliminate it?
If we do not fully understand the technology, if we are introducing it for the first time, how certain can we be that there are no risks we have yet to recognise?
Pharmaceutical history offers some uncomfortable examples. Contaminated medicines had catastrophic consequences, exposing weaknesses in supplier qualification, material verification and supply-chain oversight. Yet, at the time, the risks were not even considered. The problem was not simply a failure to follow an established procedure. It was also that the controls in place were not always sufficient to prevent or detect the danger. And sometimes, it takes a disaster before we fully recognise the weakness of a system we believed was adequately controlled. Not that a disaster is inevitable, but that we may be introducing a technology whose potential impact could develop faster than our ability to recognise its vulnerabilities.
If we are not yet certain how strong our guardrails need to be, how do we decide that they are strong enough?
Regulatory context
See the EU GMP framework (EudraLex Volume 4) and the European Commission consultation on draft Annex 22 and revised Annex 11. The proposed Annex 22 is distinguished here from the currently applicable GMP provisions.
Leave a Reply