Could a Tribunal of AI Agents Help Quality?

Have you ever noticed that when we ask AI whether an idea makes sense, we often receive a convincing, well-structured, perhaps even enthusiastic answer? But how much of that response comes from the strength of our idea, and how much from the tendency to agree with us?

This tendency is known as sycophancy: AI may favour agreement with the user over an accurate assessment. In real life, this could have serious consequences, potentially encouraging bad ideas rather than questioning them. Despite improvements in AI models, it remains an issue worth discussing. In Quality, this could become a real problem. If we ask whether the conclusion of an investigation is reasonable, we might receive a convincing justification precisely when what we actually need is an uncomfortable question.

An interesting proposal is a “tribunal” of AI agents. One agent supports the idea, another challenges it, several “jurors” examine it from different perspectives, and a judge summarises the outcome. The principle is straightforward: introduce disagreement before accepting a conclusion. However, assigning different roles does not necessarily create independent expertise. Twelve agents may share the same limitations, even if one plays the technical expert and another the Quality Manager. Secret voting might prevent agents from copying each other’s votes, but it does not eliminate shared errors.

The tribunal metaphor introduces another difficulty. If we instruct an agent to defend an idea, it may construct a convincing argument even when the evidence is weak. Conversely, an agent assigned to challenge the idea may raise irrelevant objections simply to fulfil its role. A detailed debate can appear more reliable than it actually is. For a Quality decision, the number of favourable opinions cannot replace the quality of the evidence. A single contradictory finding may be enough to challenge an entire conclusion.

To make this approach more relevant to GMP, I would change the agents’ responsibilities. One would verify original evidence; another would seek alternative explanations; a third would check applicable requirements; and a fourth would assess product and patient impact. Every observation should refer to verifiable evidence, distinguishing facts, interpretations and missing information. Initial assessments could be performed without knowing the investigator’s conclusion to reduce bias. The final summary would preserve significant disagreements and could conclude: “There is insufficient information to reach a conclusion.”

This approach is consistent with ICH Q9(R1)’s attention to bias, assumptions and relevant knowledge, but does not imply regulatory approval. Before operational use, we would need to demonstrate benefits against conventional review and a single well-instructed AI model, with appropriate validation, controls and responsibilities.

Could AI’s most valuable contribution to Quality be helping us recognise when we do not yet have enough evidence to decide?

---

So, I Created My Own Tribunal

An imaginary tribunal, created with AI assistance, examining the proposal in this article. The following are simulated specialist perspectives, not independently executed agents or experimentally verified findings.

Proposal: Use multiple AI agents to review historical GMP investigations, identify weak conclusions, challenge assumptions and recognise recurring investigation deficiencies.

Evidence available: This article, general knowledge of GMP investigation practices and the proposed architecture. No actual investigation datasets or experimental results have been provided.

1. Evidence Reviewer — Sceptical

Opinion: “The proposal addresses a plausible problem. Historical investigations can contain incomplete root-cause assessments or insufficiently supported conclusions. However, no evidence presented here demonstrates that multi-agent AI detects these weaknesses better than conventional reviews.”

Conclusion: Worth investigating, but effectiveness remains unproven.

2. Technical Reviewer — Cautiously Supportive

Opinion: “Specialised agents could separately examine original records, identify inconsistencies and compare alternative explanations. Technically feasible, although agents using similar models may reproduce identical errors. A single well-designed model could potentially achieve comparable results with less complexity.”

Conclusion: Technically plausible, but multiple agents offer no demonstrated advantage yet.

3. Regulatory Reviewer — Conditional

Opinion: “ICH Q9(R1) supports reducing subjectivity in Quality Risk Management. Existing GMP requirements also establish expectations for investigations and appropriate computerised-system controls. Nothing specifically requires or endorses an AI tribunal. Its intended use and influence on GMP decisions would determine the necessary controls.”

Conclusion: Potentially compatible with GMP principles, subject to appropriate implementation and validation.

4. Investigation Specialist — Supportive

Opinion: “The strongest application may be challenging recurring explanations such as ‘operator error’. An agent could look for overlooked equipment, procedural or organisational factors. However, an operator error can be a legitimate root cause, and the system must not automatically reject it.”

Conclusion: Potentially valuable for improving investigation depth.

5. Operational Impact Reviewer — Concerned

Opinion: “Reviewing thousands of investigations could identify trends that individual reviewers miss. But additional findings create additional work. False alarms, lengthy debates and unnecessary challenges could delay investigations without improving Quality.”

Conclusion: Benefits must outweigh additional resources and complexity.

6. Human Factors Reviewer — Sceptical

Opinion: “A tribunal may reduce the influence of the original investigator’s conclusion, but could introduce another bias: reviewers may trust a consensus between several apparently independent agents. More opinions might increase confidence without increasing accuracy.”

Conclusion: Human oversight must remain genuinely critical, not become a formality.

7. GMP Expert — Favourable to a Limited Trial

Opinion: “I would support a retrospective pilot using historical investigations, with independent SMEs assessing the findings. Compare human review, one AI model and the proposed tribunal, measuring significant omissions, false alarms and review time. Do not allow the system to approve investigations.”

Conclusion: A controlled experiment is justified; routine GMP deployment is premature.

Final Judge — Evidence-Weighted Assessment

The tribunal produced different opinions, but also identified common concerns.

Evidence: No experimental demonstration of superior performance.

Technical feasibility: Plausible, with risks of correlated errors.

Regulatory position: No specific endorsement; established GMP principles remain relevant.

Potential benefit: Better identification of unsupported conclusions and recurring investigation weaknesses.

Potential harm: False reassurance, additional workload and overconfidence in apparent consensus.

Decision: Proceed with a limited retrospective pilot, not operational deployment.

The proposal would be strengthened if independent evaluation demonstrated that multiple agents identify significant weaknesses more reliably than a single model or conventional review, without disproportionate false alarms or additional workload.

The most interesting outcome? Even when several simulated reviewers considered the proposal promising, the judge did not treat their agreement as evidence that it works.

Perhaps that is exactly what we should expect from a useful AI tribunal: not more agreement, but a better understanding of what the evidence actually supports.