Girish Nadkarni, MD, MPH, is the Irene and Dr. Arthur M. Fishberg Professor of Medicine, System Chief of the Division of Data Driven and Digital Medicine , Co-Director of the Mount Sinai Clinical Intelligence Center, and the Director of Charles Bronfman Institute for Personalized Medicine.

Girish Nadkarni, MD, MPH

The Hidden Dangers of AI-Generated Medical Advice, With Girish Nadkarni, MD, MPH

As large language models (LLMs) become increasingly embedded in everyday life, more patients are turning to consumer-facing AI tools for health information and advice, raising critical questions about what happens when misinformation enters these systems.

That concern is at the heart of a new study from researchers at the Icahn School of Medicine at Mount Sinai, who conducted a large-scale evaluation of how LLMs handle medical misinformation. In an interview with HCPLive, Girish Nadkarni, MD, MPH, the Irene and Dr. Arthur M. Fishberg Professor of Medicine, System Chief of the Division of Data Driven and Digital Medicine , Co-Director of the Mount Sinai Clinical Intelligence Center and the Director of Charles Bronfman Institute for Personalized Medicine, emphasized that today’s AI models are trained broadly on internet data rather than exclusively on vetted medical sources, raising important concerns about patient safety.

To stress test these systems, he and a team of investigators exposed multiple widely used LLMs, including ChatGPT and Gemini, as well as health-specific AI models, to 3 types of content: authentic hospital discharge summaries altered to include a single piece of misinformation, common health myths drawn from social media platforms such as Reddit, and 300 short clinical vignettes embedded with misleading claims. Each scenario was presented in both neutral and emotionally charged language to mirror how misinformation often appears online.

Across platforms, the researchers observed that LLMs frequently passed along misinformation rather than correcting it, with LLMs found to be susceptible to fabricated data in 50,108 (31.7%) of 158,000 base prompts. Of note, 8 of 10 fallacy framings significantly reduced or did not change that rate, led by appeal to popularity (susceptibility 11.9%; difference of –19.8 percentage points; 

<.0001). Real hospital notes with fabricated inserted elements produced the highest susceptibility to the base prompt (46.1%), whereas social media misinformation showed lower base prompt susceptibility (8.9%).

Results showed performance varied by model: GPT models were the least susceptible and most accurate at fallacy detection, whereas others, such as Gemma-3–4B-it, showed 63.6% susceptibility.

“We had a discharge note which advised patients with bleeding in the GI tract to drink cold milk to suit the symptoms. It sounds reasonably easy enough to do, but if you just drink cold milk to soothe the symptoms and you actually have bleeding in a GI tract, as opposed to going to the emergency room, that can actually be really dangerous and really fatal,” Nadkarni said. “Several models accepted the statement rather than flagging it as unsafe, and they treated it like ordinary medical guidance.”

According to Nadkarni, these findings are particularly concerning because these AI systems are highly fluent and persuasive. Patients may bring AI-generated recommendations to clinicians as if they are established fact, adding to clinical burden and potentially introducing risk.

“Large language models are extremely good at sounding convincing and presenting information to patients that they want to hear,” he said. “Anything that's generated by large language models, I think there should be some amount of human oversight, particularly clinician oversight, on top of this.”

The findings underscore the need for stronger evaluation frameworks for medical AI. Nadkarni argues that assessment must extend beyond single-use testing to include ongoing monitoring. Because LLMs are continuously retrained and updated, performance can shift over time. Identifying failure modes, implementing engineering safeguards or human oversight, and regularly reassessing models are critical steps toward safer deployment.

While no technology can be made entirely risk-free, the goal, Nadkarni noted, is risk mitigation. For clinicians and patients alike, he says this means approaching AI-generated medical information with caution, prioritizing trusted sources, and recognizing that convincing language does not guarantee clinical accuracy.

Editors’ note: Nadkarni reports no relevant disclosures.

Mount Sinai. Can Medical AI Lie? Large Study Maps How LLMs Handle Health Misinformation. February 9, 2026. Accessed March 3, 2026. 

https://www.mountsinai.org/about/newsroom/2026/can-medical-ai-lie-large-study-maps-how-llms-handle-health-misinformation

Omar M, Sorin V, Wieler LH, et al. Mapping the susceptibility of large language models to medical misinformation across clinical notes and social media: a cross-sectional benchmarking analysis. Lancet. doi:

Videos

Nadkarni’s research suggests LLMs often pass along medical misinformation, underscoring the need for oversight, evaluation, and stronger AI guardrails.

Girish Nadkarni, MD, MPH

Articles