Medical AI systems often struggle with difficult questions even when they use multiple agents to debate possible answers. A new study proposes adapting those teams as reasoning unfolds: an external “Challenger” reviews the discussion, identifies specific gaps, and helps guide targeted evidence retrieval or the addition of a specialist.
The researchers evaluated the approach on 862 difficult questions from nine biomedical datasets in the MedAgentsBench benchmark. Retrieving evidence before discussion began offered little benefit, while retrieving it between reasoning rounds in response to an identified gap performed better. On a multi-specialist system, this method improved accuracy on eight of the nine datasets and scored six percentage points higher than using the Challenger without retrieval.
The study also examined whether a team should recruit another specialist when it encounters a gap in its expertise. Self-triggered recruitment—where the agents decide for themselves that another perspective is needed—was unreliable. Challenger-informed recruitment improved GPT-4.1-mini’s accuracy on MedQA from 46% to 54%, with results moving in a positive direction across the other tested generalization datasets.
Combining evidence retrieval and specialist recruitment without coordination reduced performance. The authors say this points to the importance of managing how the two interventions interact, rather than simply adding both to a system. Their results suggest that the timing and reason for bringing in outside knowledge or expertise are important design choices for medical multi-agent reasoning.
The work uses multiple-choice medical questions as a controlled test and notes that the benchmark does not fully represent decision-making in a hospital. The Challenger identifies weaknesses in the discussion but does not provide the missing medical information itself; separate components handle retrieval or specialist recruitment.
Sources: arXiv cs.AI.