Traditional Diagnostic Support Systems Surpass Generative AI in Disease Diagnosis

In the evolving landscape of medical diagnostics, traditional expert systems have demonstrated superior performance compared to generative AI models. Historically, diagnostic decision support systems (DDSSs), such as the well-established DXplain developed by Massachusetts General Hospital in 1984, have utilized extensive databases of disease profiles and clinical findings to assist clinicians in identifying potential diagnoses. These systems have been integral in augmenting medical decision-making by recalling relevant information rapidly, reducing cognitive load, and minimizing diagnostic errors.

With recent advances in artificial intelligence, particularly the advent of large language models (LLMs) like ChatGPT and Gemini, researchers at MGH’s Laboratory of Computer Science undertook a comparative study to evaluate their diagnostic accuracy against DXplain. The study involved 36 diverse patient cases across different demographic groups, assessing each system’s ability to recommend correct diagnoses both with and without laboratory data.

The findings revealed that DXplain had a slight edge, correctly identifying diagnoses 72% of the time with lab data and 56% without, outperforming LLMs which achieved 64% and 42%, respectively. Nonetheless, LLMs performed commendably and showed potential when integrated with traditional systems. Experts suggest that combining the explanatory strengths of DDSSs with the linguistic and narrative capabilities of modern LLMs could significantly enhance clinical diagnostic support and improve patient outcomes.

Dr. Edward Hoffer emphasized the continued importance of expert systems, highlighting their bias-free recall of medical knowledge and unbiased reasoning. Meanwhile, Mitchell Feldman underscored the promise of hybrid approaches, where LLMs can extract clinical findings from narrative text for input into DDSSs. This synergy could overcome limitations inherent in each individual technology and facilitate more accurate, rapid diagnosis in complex cases.

Overall, while generative AI models are advancing, traditional diagnostic systems remain valuable tools, and their integration with AI could represent the future of clinical decision support systems, ultimately leading to better diagnostic accuracy and patient care.