Limitations of AI in Emergency Room Diagnoses Based on Symptom Presentation

Artificial intelligence (AI) has shown promise in supporting emergency room physicians with disease diagnosis, but its effectiveness remains limited to cases exhibiting typical, well-defined symptoms. Researchers from West Virginia University conducted a comprehensive study comparing different iterations of ChatGPT models, including GPT-3.5, GPT-4, and their variants, in diagnosing patients based on physicians' notes. The study, published in Scientific Reports, revealed that current AI models tend to perform adequately when patients present classic symptoms; however, they struggle with atypical or challenging cases that do not conform to textbook presentations.

The researchers analyzed data from 30 anonymized emergency department cases, focusing on how often AI could correctly identify the top diagnosis. Results indicated that while newer models show a modest improvement in accuracy for the primary diagnosis—about 15% to 20% higher than previous versions—the overall ability to accurately diagnose complex or atypical cases remains limited. For example, in cases of pneumonia without the typical fever, all tested models failed to produce the correct diagnosis in their top three suggestions.

Lead researcher Gangqing "Michael" Hu emphasized the importance of incorporating diverse and comprehensive data sources—such as laboratory results and imaging—to enhance AI diagnostic accuracy in the future. He also highlighted that, despite AI's potential, it is not a certified medical device and should be used with human oversight in clinical settings.

The team envisions future developments involving multi-agent conversational AI systems that interact and reason collectively, thereby improving trust and transparency in AI-driven diagnostics. Hu noted that current AI tools are best suited as assistive systems rather than sole decision-makers and stressed the importance of continuous research to expand AI’s capabilities beyond standard presentations.

Ultimately, Hu advocates for cautious integration of AI into emergency medicine, emphasizing that accuracy for challenging, atypical cases still requires human judgment to ensure patient safety and high-quality care.

Source: https://medicalxpress.com/news/2025-05-ai-usefulness-emergency-room-limited.html