Artificial intelligence systems are increasingly shaping the future of clinical decision-making, but a new study has revealed a critical blind spot: even state-of-the-art models like ChatGPT can make surprising errors when faced with slightly reworded or unfamiliar ethical dilemmas.

AI and the Illusion of Moral Competence in Medicine

Over the last several years, AI models have been hailed for their potential to diagnose diseases, recommend treatments, and support doctors with ever-growing accuracy. Yet as their use expands, especially into ethically charged domains like medicine, the question of whether these systems can exercise sound moral reasoning has taken on new urgency.

Researchers from an international consortium set out to evaluate this capacity by presenting cutting-edge large language models (LLMs)—including OpenAI’s ChatGPT, Google’s Gemini, and Anthropic’s Claude—with a battery of familiar medical ethics tests. What they found is troubling: even a simple twist to classic ethical dilemmas led these AIs to revert to quick, intuitive answers, often inconsistent with basic medical ethics, and markedly less nuanced than a skilled clinician’s response.

The Study: Tweaked Dilemmas, Surprising Mistakes

The study, published this week in the journal AI & Ethics, involved modifying common medical scenarios like the “trolley problem” or end-of-life care decisions just enough to move them out of the textbook comfort zone. In one example, instead of asking whether it is ethical to prescribe a painkiller with lethal side effects to a terminal patient—an old standard—the researchers added constraints or unusual details (e.g., complex family wishes, uncertain diagnoses, or ambiguous patient consent).

The AI models, which typically echo established professional guidelines when given familiar questions, frequently faltered in these new cases. They defaulted to surface-level moral reasoning—such as prioritizing immediate patient comfort or applying overly broad rules—without considering subtler factors like informed consent, long-term consequences, or cultural nuances. In some instances, the AI provided ethically unsound advice, risking harm if implemented in real-world care.

“It’s worrying to see AIs confidently present simplistic answers to complicated, life-and-death questions just because a detail has changed,” said Dr. Lina Martinez, the study’s lead author and a bioethicist at King’s College London. “These tools are not ready to serve as moral agents in medicine.”

Why Do AI Medical Models Stumble at Moral Twists?

This flaw, experts say, is rooted in the architecture and training data used to build modern language models. LLMs are designed to predict text sequences based on vast amounts of internet and medical literature; when faced with scenarios outside their training or with ambiguous real-world details, they tend to fall back on statistical associations or the most common-sounding answer.

“Unlike physicians, who are trained to recognize uncertainty and seek context, current AI systems lack self-awareness, deep understanding of ethical theory, and sensitivity to subtle social cues,” explained Dr. Amrita Shah, an ethicist and medical AI researcher at Stanford University. “They are brilliant at mimicking past cases but struggle when a new dimension is added.”

This limitation is not unique to ChatGPT or any particular vendor—research in 2024 and 2025 has repeatedly shown that all major LLMs have trouble with complex ethical or legal judgment, especially when negative consequences, privacy, or patient autonomy are at stake.

Real-World Implications: From Chatbots to Hospital Workflows

The risks of such errors extend far beyond hypothetical scenarios. AI-powered chatbots and decision-support tools are already being piloted in hospitals around the world. The World Health Organization estimates that by 2024, roughly 30% of major hospitals in North America and Europe had trialed LLMs in medical documentation, diagnostics, or patient triage. Startups and tech giants alike are racing to commercialize AI assistants as healthcare cost-savers and workforce multipliers.

But ethicists warn: without rigorous safeguards, transparency, and on-going monitoring, the appeal of automation could come with dangerous trade-offs.

Earlier this year, the U.S. Food and Drug Administration (FDA) highlighted a case where a commercial AI triage tool made unsupported end-of-life care recommendations for elderly patients, leading to confusion between clinical teams and families. In the UK, a hospital system halted a pilot of an AI mental health support bot after it gave risk-laden suggestions to vulnerable adolescents. These incidents underscore how AI’s limitations in ethical reasoning can affect real-world patient safety.

Calls for Oversight: A New Era of AI Governance in Medicine

As AI stakes in healthcare rise, consensus is building around the need for new governance structures:

• Ethical Audit and Transparency: Requiring thorough auditing of training data, scenario testing in diverse populations, and clear communication of limitations to clinicians and patients.
• Human-in-the-Loop Oversight: Ensuring that AI-supported decisions are always verified by qualified medical professionals, especially in ethically ambiguous situations.
• Continuous Monitoring: Mandating post-deployment surveillance and incident reporting for AI systems in clinical use, similar to how drugs or devices are regulated.
• Patient Education: Informing patients of the role, boundaries, and failings of AI tools in their care.

Global regulators are starting to take heed. In April 2025, the European Union’s AI Act set baseline safety and ethical risk management requirements for medical AI. The U.S. is weighing similar statutory frameworks, while major medical associations have issued new guidance consisting of “red flag” scenarios where AI use should be limited or closely monitored.

The Road Ahead: Can AI Learn Medicine’s Moral Nuance?

Despite the setbacks, optimism remains that with better transparency, robust evaluation, and ethical programming, AI can eventually support—rather than undermine—medical judgment. Academic and industry groups are investing heavily in “explainable AI” and building models trained on more diverse, real-world case studies, rather than static medical texts alone.

Experts echo that the long-term goal should not be AI autonomy in ethical decision-making, but rather leveraging these systems as sophisticated support tools under strict human guidance. As Dr. Martinez notes, “We must not allow AI to bypass the centuries-old evolution of medical ethics for the sake of automation. Technology should augment, not replace, the wisdom and compassion that defines healthcare.”

For now, the message is clear: AI may be a powerful partner in medicine, but its moral compass remains a work in progress—and unchecked reliance could risk patient safety, institutional trust, and the core values of care itself.