Collaborative AI Achieves Top Scores in U.S. Medical Licensing Exams

Researchers have developed an innovative approach using a council of AI models that collaboratively discuss and refine their answers, leading to unprecedented performance on U.S. medical licensing exams (USMLE). This collaborative AI system, composed of multiple instances of OpenAI's GPT-4, engaged in coordinated and iterative exchanges to reach consensus responses. In a recent study published in PLOS Medicine, this method achieved accuracy rates of 97%, 93%, and 94% on Step 1, Step 2 CK, and Step 3 exams respectively, surpassing individual GPT-4 performance.

The study highlights how collective decision-making among AI agents can enhance diagnostic accuracy, which is crucial in healthcare. When faced with 325 publicly available USMLE questions covering biomedical sciences and clinical diagnosis, the council demonstrated the ability to self-correct, correcting over half of the responses that were initially incorrect based on majority voting.

This approach addresses common issues in language models, such as inconsistent responses and hallucinations, by utilizing group deliberation. The authors suggest that such collaborative AI could serve as more reliable tools for medical education and, potentially, clinical decision-making in the future. However, they underscore that this paradigm has yet to be tested in real-world clinical settings.

The study emphasizes that embracing variability and teamwork in AI responses can unlock new possibilities for medical applications, promoting higher accuracy and trustworthiness. As noted by researchers Shaikh, Siddiqui, and Asiyah, collaborative dialogue among AI systems can lead to improved self-correction and performance, offering a promising avenue for advancing AI in healthcare.

Source: https://medicalxpress.com/news/2025-10-collaborative-ai-medical-exams.html