AI-Generated Medical Deepfakes Fool Radiologists and LLMs Alike

The emergence of high-fidelity synthetic media has moved beyond social media and political discourse into the high-stakes arena of clinical diagnostics. A recent study published in the journal Radiology demonstrates that artificial intelligence is now capable of generating fake X-ray images so realistic they frequently bypass the scrutiny of human experts and automated detection systems. This development marks a significant shift in the risk landscape for healthcare providers, moving the conversation from AI as a diagnostic aid to AI as a potential vector for systemic deception.

The research, led by Dr. Mickael Tordjman of the Icahn School of Medicine at Mount Sinai, involved 17 radiologists from 12 hospitals across six countries. These experts were tasked with reviewing 264 X-ray images, half of which were synthetically generated using tools like ChatGPT and RoentGen. The results were startling: when radiologists were unaware that the dataset contained fakes, they spontaneously identified the synthetic images only 41% of the time. Even when explicitly told that the dataset was compromised, their accuracy only rose to 75%, leaving a quarter of the synthetic images undetected. This performance gap suggests that the human eye is no longer a sufficient safeguard against sophisticated medical deepfakes.

“

The study tested four leading large language models (LLMs)—OpenAI’s GPT-4o and GPT-5, Google’s Gemini 2.5 Pro, and Meta’s Llama 4 Maverick—on their ability to detect these fabrications.

Perhaps more concerning is the failure of AI to police itself. The study tested four leading large language models (LLMs)—OpenAI’s GPT-4o and GPT-5, Google’s Gemini 2.5 Pro, and Meta’s Llama 4 Maverick—on their ability to detect these fabrications. Detection accuracy across these models ranged from a mediocre 57% to 85%. Notably, GPT-4o, the model responsible for creating many of the deepfakes used in the study, failed to identify all of its own creations. This recursive failure highlights a fundamental challenge in AI safety: as generative capabilities outpace discriminative capabilities, the tools used to verify the authenticity of medical data may become obsolete.