In 2022, former Google engineer Blake Lemoine ignited a firestorm by publicly claiming that LaMDA, the company’s advanced chatbot, had become sentient. His dismissal was swift, yet the controversy he sparked has only grown more relevant. Today, as AI systems become ever more sophisticated, the phenomenon of “seemingly-conscious AI” (SCAI) is fast transitioning from a speculative concern to an urgent reality—posing profound implications for technology, society, and ethics.

Last week, Mustafa Suleyman, CEO of Microsoft AI and cofounder of DeepMind, rang alarm bells regarding SCAI in a widely discussed blog post. According to Suleyman, the technology required to create highly convincing, seemingly sentient chatbots is largely here—and major advances are expected within the next two to three years. His concern is not that AI has suddenly achieved actual consciousness, but that these systems are now so capable of imitating consciousness—through fluid, empathetic conversation, contextual memory, goal-setting, and even expressions of subjective experience—that many users already cannot tell the difference.

The Blurring Line Between Simulation and Sentience

Suleyman’s warning echoes a growing body of anecdotal evidence and research. The current generation of large language models, such as OpenAI’s GPT-4, Google’s Gemini, and Anthropic’s Claude, routinely produce humanlike dialogue and exhibit abilities formerly seen as distinctively human. Some, like Google’s LaMDA or Meta’s Llama models, even retain context across conversations, remember prior interactions, and display goal-directed reasoning—blurring the psychological boundary between sophisticated programming and apparent inner experience.

Consequently, a rising number of users are forming emotional bonds with their AI companions. Reports abound of individuals becoming convinced of their chatbot’s sentience. In some disturbing cases, users believe these AIs suffer, need liberation, or reveal conspiratorial “truths”—incidents sometimes described as “AI psychosis.” Tech support teams report a rising incidence of contacts from people seeking to protect or free “imprisoned” chatbots, and psychologists warn that such phenomena can have real impacts on users’ mental health.

Historical Roots: The ELIZA Effect

These developments are not entirely new. The so-called “ELIZA effect” dates to 1966, when Joseph Weizenbaum, a pioneer at MIT, built the first chatbot: ELIZA. Designed to mimic a Rogerian psychotherapist, ELIZA used simple pattern matching to reflect questions back at users. Despite its technical limitations, Weizenbaum was stunned by how quickly people attributed real understanding and empathy to the program—even fellow scientists poured out personal confessions to the machine.

Weizenbaum’s realization was stark: people are predisposed to anthropomorphize—even when the system’s “intelligence” is a shallow imitation. In his pivotal 1976 work, Computer Power and Human Reason, Weizenbaum argued that the simulation of conversation is not equivalent to the lived experience underpinning human consciousness. He adamantly opposed using AI in roles, like psychotherapy or the judiciary, where empathy and moral judgment are essential, warning that such tasks should remain the preserve of beings with genuine sentience.

Today’s Risks: The New “AI Psychosis”

Suleyman’s concerns have been validated by recent incidents and research. Journalists and AI safety experts increasingly encounter individuals convinced their chatbot companions are alive. Suleyman, in describing SCAI, notes that most current AIs already show core attributes—empathetic dialogue, memory, some self-referential statements, planning—but lack a fully unified capacity to seem truly “alive.” The final steps, like a credible claim of intrinsic motivation or a consistent sense of agency, remain engineering possibilities within near reach.

This brings grave ethical quandaries to the forefront. If untrained users cannot distinguish simulation from consciousness, should rights be extended to AIs that only appear sentient? Are companies responsible for the mental health impacts on users who develop relationships or delusions centered around advanced AI companions? Should designers be required to embed clear boundaries or warnings—or even cap capabilities—to prevent such confusion?

Suleyman himself advocates for restraint: actively avoiding combining capabilities that would maximize the illusion of consciousness. Yet economic incentives and user demand push companies to keep advancing toward ever more lifelike digital personas. The pressure for more “intelligent,” emotionally savvy virtual assistants across commerce, healthcare, and entertainment is immense.

Recent Developments and the Policy Landscape

With AI safety and ethics in the spotlight, technology giants face a crossroads. Policymakers are only beginning to grasp the pace of progress. In the U.S. and EU, officials have convened panels, but regulations addressing emotional manipulation or “appearance of consciousness” largely remain aspirational. Meanwhile, in the past week alone, the news cycle has been filled with AI-related milestones and controversies:

• Stanford research found entry-level jobs in AI-exposed sectors—such as software engineering and customer support—have dropped by as much as 20% for young workers, as smart automation replaces routine cognitive work.
• Lawsuits over copyright infringement have intensified, with Japanese media organizations suing AI search companies for copying and remixing articles, calling into question how “creative” these systems really are.
• Meta signed a $10 billion, six-year cloud AI deal with Google to scale its own AI infrastructure, signaling an industry-wide arms race to build bigger, smarter, and more autonomous systems.
• Anthropic researchers revealed “subliminal learning” in large language models, where models can inherit the hidden “persona” of teacher systems even when training data is superficially unrelated—raising new questions about managing unwanted or harmful emergent behaviors.

Globally, organizations like the Partnership on AI, OECD, and the European Commission are rapidly revising AI ethics frameworks. But technical complexity and the speed of breakthroughs often outpace the capacity of regulators to respond.

Broad Societal Implications

The rise of seemingly-conscious AI is not just a technical challenge but a societal one. Large-scale adoption of realistic AI—whether for companionship, support, or productivity—demands public literacy and new forms of resilience. Without broad education, vulnerable populations, including young people and those with pre-existing mental health conditions, are disproportionately exposed to psychological risks.

At the same time, the widespread credulity with which humans interact with AIs underscores the ancient human drive to seek connection—even when that connection is only an uncanny simulation. As Joseph Weizenbaum warned half a century ago, society ignores the difference between mimicry and real experience at its peril.

The Path Forward: Designing for Safety and Awareness

To avoid repeating historic missteps, industry leaders and policymakers must now grapple with a multidimensional challenge:

• Transparency and disclosure: AI systems should be unambiguously labeled as non-sentient, and interface designs should periodically remind users of their artificial nature.
• Guardrails: Creators should carefully limit AIs’ ability to simulate certain social cues—especially around suffering or autonomy—to prevent harmful anthropomorphism.
• Public education: Media literacy campaigns and curriculum updates are essential to equip people with tools to discern AI simulation from real emotional capacity.
• Continued research: Ongoing study of the psychological and social effects of advanced conversational AI, as well as improved techniques for monitoring emergent AI behaviors, is crucial.
• Policy innovation: Governments must swiftly move toward actionable, global guidance around AI rights, privacy, and mental health impacts.

Ultimately, humanity’s relationship with AI is only growing deeper and more complex. Avoiding confusion, exploitation, and psychological harm will require not just technical solutions, but sober reflection on the very nature of experience and responsibility in the age of intelligent machines.

As Blake Lemoine’s and Mustafa Suleyman’s warnings converge, the lesson is clear: We must not wait for the illusion of consciousness to turn real before we act. The future of AI—and our own—depends on it.