Groundbreaking Evidence-Based AI Outperforms Most Doctors on USMLE Exams

A novel clinical artificial intelligence tool developed by researchers at the University at Buffalo has shown exceptional accuracy on all three parts of the United States Medical Licensing Examination (USMLE), according to a recent publication in JAMA Network Open. This AI, named Semantic Clinical Artificial Intelligence (SCAI), surpasses existing AI models and most physicians in exam performance, with the most advanced version achieving an impressive score of 95.2% on Step 3—compared to 90.5% scored by GPT-4 Omni.

SCAI is designed to serve as a decision-support tool that can augment physicians' reasoning and clinical decision-making, moving beyond simple data retrieval. Unlike typical generative AI, which primarily relies on statistical associations and may produce answers based on memorized content, SCAI employs semantic reasoning and expansive authoritative data sources. It incorporates 13 million medical facts and their interactions, constructed through semantic networks and knowledge graphs. This extensive knowledge base allows it to draw logical inferences and provide contextually accurate responses.

The development process involved enhancing natural language processing software with diverse medical sources, including current literature, clinical guidelines, genomic data, drug information, discharge summaries, and safety data, while excluding biased data like clinical notes. Techniques such as retrieval-augmented generation and formal semantics underpin SCAI’s sophisticated reasoning capabilities.

Uniquely, SCAI can engage in conversational interactions, adding a human-like quality to its responses and supporting complex, evidence-based medical reasoning. Its potential extends to improving patient safety, increasing access to specialized care, and supporting primary care providers and patients with easily accessible, reliable medical information.

While SCAI represents a significant technological advancement, experts emphasize that it is intended to complement, not replace, physicians. Elkin, the lead researcher, states, “A doctor who uses AI may replace a doctor who does not,” underscoring the transformative role of AI in medicine. The study highlights the importance of integrated semantics and reasoning in AI development, pointing to a future where such tools enhance clinical practice and decision-making.

The full details of the research can be found in the publication: Semantic Clinical Artificial Intelligence vs Native Large Language Model Performance on the USMLE, in JAMA Network Open, 2025.