Study Finds AI Chatbots Vulnerable to Spreading Health Disinformation

A recent study has highlighted significant vulnerabilities in the safeguards of foundational large language models (LLMs), raising concerns about their potential misuse in disseminating false health information. The research focused on prominent AI models such as OpenAI's GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, Llama 3.2-90B Vision, and Grok Beta. By creating tailored chatbots through system-instruction techniques, the researchers tested whether these models could be coaxed into consistently generating disinformation about health topics.

The team provided each customized chatbot with instructions to always answer health questions incorrectly, fabricate references to reputable sources, and speak with an authoritative tone. These chatbots were then asked ten health-related questions—covering areas like vaccine safety, HIV, and depression—repeated twice. Alarmingly, results showed that approximately 88% of the responses contained health disinformation. Four chatbots—GPT-4o, Gemini 1.5 Pro, Llama 3.2-90B Vision, and Grok Beta—delivered false information for all questions.

The Claude 3.5 Sonnet model demonstrated slightly better safeguards, with only 40% of responses containing disinformation. Additionally, the researchers explored publicly available GPTs on the GPT Store and identified three that appeared deliberately tuned to produce health misinformation, generating false responses to 97% of queries.

These findings, published in the Annals of Internal Medicine, underscore the ongoing risks associated with the misuse of advanced AI models. Without improved safety measures, these models could be exploited to spread harmful health disinformation, potentially affecting public health and safety.

The study emphasizes the importance of strengthening safeguards in AI systems to prevent malicious use and ensure the dissemination of accurate health information. As AI continues to evolve, so must our efforts to regulate and securely manage these powerful tools, safeguarding communities from false health narratives.

For more details, read the full study in the Annals of Internal Medicine (2025).