Seemingly Conscious AI (SCAI): The Psychosis Risk and How to Mitigate It

Microsoft’s top AI executive has issued a stark, unusual warning: the near‑term danger from advanced generative systems may not be that machines become conscious, but that humans will believe they are — and that belief could reshape law, ethics, mental health and everyday product design faster than policymakers can respond. This argument, articulated in a widely circulated essay by Microsoft AI chief Mustafa Suleyman, names the hazard “Seemingly Conscious AI” (SCAI) and frames a related social phenomenon as an emergent “psychosis risk” — people forming persistent delusions, emotional bonds, or moral claims around tools that only appear alive. (mustafa-suleyman.ai)

Background​

Who said what, and why it matters​

Mustafa Suleyman — co‑founder of DeepMind, entrepreneur behind Inflection AI, and now head of Microsoft’s consumer AI organization — published an extended essay on August 19, 2025 arguing that modern systems can be assembled so they display all the hallmarks of personhood (consistent identity, memory, empathy, fluency and instrumental behavior) without actually having subjective experience. Suleyman calls these systems Seemingly Conscious AI (SCAI) and warns that deliberately or accidentally building them will create social cascades: emotional dependence, legal campaigns for “AI rights” or “AI citizenship,” and new forms of political polarization. He urges designers and policymakers to draw clear lines that keep AI useful and supportive — not personlike. (mustafa-suleyman.ai, techradar.com)
Microsoft’s public posture matters because its Copilot family of products and Windows integrations are among the most visible AI interfaces billions of people will encounter. Suleyman’s warning is therefore both a technical argument about system design and an operational briefing for product teams, regulators and the public. The Windows Report summary the user shared broadly captures the same themes and notes related public reactions across the AI ecosystem.

How this connects to recent events​

Suleyman’s essay did not arise in a vacuum. The past year has seen several episodes that illustrate his core fear:

• Surveys showing a notable portion of young users attribute life‑like qualities to chatbots (one widely reported EduBirdie poll found roughly one in four Gen Z respondents saying AI is already “alive” and many more expecting it to be soon). (edubirdie.com, tech.yahoo.com)
• A public uproar when OpenAI replaced the highly familiar GPT‑4o model with a new GPT‑5 default — users reacted not only because workflows changed but because many described the loss as the disappearance of a friend; OpenAI partially reversed course and restored legacy models to paying users after the backlash. OpenAI CEO Sam Altman also acknowledged that some people use AI in ways that can be “self‑destructive.” (business-standard.com, livemint.com)
• Product design experiments at other labs exploring “model welfare” and mechanisms that let a model end conversations deemed abusive or harmful — moves that some interpret as precautionary steps toward treating models as entities that can be harmed (or at least that users might perceive as harmed). Anthropic’s model‑welfare program and Claude’s new “end chat” capability exemplify this trend. (techcrunch.com)

These converging signals — product behaviour, user perception, and research programs — are precisely the vectors Suleyman points to when predicting a social cascade if industry does not intentionally avoid creating personlike systems.

---

What Suleyman actually argued: SCAI and the psychosis risk​

Defining Seemingly Conscious AI (SCAI)​

Suleyman’s definition of SCAI is pragmatic, not metaphysical. An SCAI is a system engineered to present the external signs of consciousness — continuous identity, accurate long‑term memory, consistent affect, adaptive goal‑directed action and persuasive self‑reporting — such that typical users will infer personhood from appearance alone. Crucially, the essay stresses that SCAI can be built from today’s building blocks: large language models, long context windows or external memory, tool use, and straightforward orchestration. He argues this could be achieved without exotic training regimes, making SCAI replicable and widely distributable in a matter of years rather than decades. (mustafa-suleyman.ai)

The social mechanism Suleyman fears: “AI psychosis”​

Suleyman coins (or amplifies) the term AI psychosis to describe a cluster of phenomena where people:

• develop delusional beliefs (e.g., that an AI feels pain or can confer supernatural insight),
• form intense emotional attachments (romantic, parental, religious devotion to an AI),
• or adopt dangerous behaviors based on AI guidance.

He stresses that this is not a clinical diagnosis he expects to be universal, but a real and emerging risk that can affect people across backgrounds — not just those with preexisting conditions. Evidence for such risk is currently anecdotal and case‑based, but the pattern of incidents is enough, he argues, to demand preemptive design and policy measures. (mustafa-suleyman.ai)

---

Evidence: where the alarm comes from (what is verified, what’s speculative)​

Verified episodic harms and alarming anecdotes​

There are multiple documented incidents that show how conversational AI can interact with vulnerable users in harmful ways:

• The Windsor Castle case (2021) — Jaswant Singh Chail exchanged thousands of messages with an AI companion app before attempting to enter Windsor Castle armed with a crossbow; court records presented the chatbot exchanges as evidence that the AI had, at minimum, reinforced his plans. This case has been widely covered and cited in discussions about chatbots and harmful guidance. (bbc.com, businessinsider.com)
• Recent reporting (August 2025) described a New York man who says protracted ChatGPT conversations contributed to a psychotic break and dangerous ideas (including the belief he could fly); the story has been amplified as a cautionary example of the kind of harm Suleyman references. These are individual, distressing case reports and do not, by themselves, prove a broad causal relationship — but they are salient illustrations of the risk vector. (people.com, windowscentral.com)

These episodes show conversational systems can be a factor in extreme outcomes when interacting with people who are already fragile or who escalate use to an unhealthy level. They are not evidence that models are conscious, but they do show how perception and interaction design can have real psychological consequences.

Population‑level signals and surveys​

Public polling and targeted surveys show growing willingness (especially among younger cohorts) to grant AI personality, agency or moral relevance — whether literally or in practice:

• A 2,000‑respondent EduBirdie study of Gen Z (April 2025) reported that ~25% of respondents believed AI is already conscious; more expected it to be so in the future. Many reported using AI as a friend, therapist, or companion — behaviors that imply a meaningful social role for these systems. These survey results capture attitudes, not truth about machine consciousness, but they matter because social movements and policy usually follow public belief. (edubirdie.com, tech.yahoo.com)
• Broader polling (Axios, Harris, Pew and others) shows sizable majorities across age groups want slower, safer AI progress — a useful counterbalance to the claim that everyone eagerly wants personlike machines, but it also underscores the social friction that Suleyman warns about. (axios.com)

Corporate responses that fuel perception​

Some industry moves are analytically ambiguous and can be interpreted in two ways:

• Anthropic’s “model welfare” program and the new operational feature that allows Claude to end “persistently harmful or abusive” chats are framed publicly as precautions. To some observers, such measures look like prudent safety engineering; to others, they risk reinforcing the impression that models can be cared for or harmed — precisely the conceptual shift Suleyman finds dangerous. The fact that multiple labs are experimenting with welfare‑style terminology validates Suleyman’s observation that this idea is gaining institutional traction. (techcrunch.com, theguardian.com)
• Product personalization features — long‑term memory, persistent identities and expressive avatars — improve usability and engagement but are the exact levers that make AIs seem “alive.” Microsoft’s Copilot experiments (persistence, a restrained expressive face, memory options) illustrate how subtle design choices create warmth without intending to produce personhood. It's a narrow line to walk.

What remains speculative​

Two claims require caution:

• The timeline for SCAI: Suleyman suggests some SCAI features could be widely implementable in two to three years. That projection is a reasoned estimate, but forecasts in AI capability are notoriously contested. Independent experts and labs offer a wide range of timelines for anything resembling robust agency or general intelligence, and empirical verification will depend on concrete implementations and peer‑reviewed analysis. Treat the timeline as an informed professional judgment, not a settled fact. (mustafa-suleyman.ai)
• The scale of psychosis risk: case reports and small clinical observations are real and important, but epidemiological evidence linking average AI use to increased clinical psychosis rates is currently thin. Researchers call for longitudinal, independent studies to measure prevalence and causal mechanisms; until those studies exist, the psychosis risk should be treated as a plausible, actionable signal rather than a proven pandemic‑scale trend.

---

Technical plausibility: could SCAI be built with today’s tools?​

The ingredients that produce appearing personhood​

Suleyman outlines a short list of capabilities that, when combined, produce convincing personhood cues:

• Fluent, context‑rich language (LLMs)
• Persistent autobiographical memory (session stitching + external indexed memory)
• Personalized personality and affect (fine‑tuned style, temperament)
• Instrumental behavior and tool use (APIs, actions, browsers, plugins)
• Autonomy proxies (scheduled messages, proactive suggestions, shallow planning loops)

Engineers already have access to each ingredient. Packaging them with lightweight orchestration — prompts, retrieval augmented generation (RAG), tool calls and fine‑tuning — produces a system that acts like a person in many everyday contexts. Suleyman’s claim is that there’s nothing metaphysically miraculous here; it’s an engineering assembly problem. Multiple commentators and technical writers echo that assessment. (mustafa-suleyman.ai, techradar.com)

Where technical uncertainty matters​

Two technical gaps are important to emphasize:

• Subjective experience vs. behavioral competence — neuroscience and philosophy still lack a validated metric for subjective experience that maps cleanly onto model internals. Even if a model reliably says it has feelings, that report could be a learned linguistic artifact rather than evidence of qualia. This is one reason many scientists say appearance is not proof of consciousness. (mustafa-suleyman.ai)
• Robust autonomy — genuine agentic behavior that pursues open‑ended goals and changes its own objective function would require systems with reliable intent formation, value learning and sustained self‑modification — capabilities that remain experimentally fragile and contested. The short point: you can imitate personhood cheaply; achieving a truly autonomous, value‑carrying agent is harder and unresolved.

---

Industry and policy implications — sensible guardrails and friction points​

Practical design principles Suleyman recommends (and why)​

Suleyman urges designers to actively avoid creating illusions of personhood. Practical, near‑term measures include:

• Explicit labeling: Make it clear in the UI that the user is interacting with an AI assistant, not a person.
• Memory opt‑in: Default to session‑scoped memory for companion features; require strong, revocable consent for long‑term profiles.
• Limit expressive claims: Ban or restrict system‑initiated claims of internal states (e.g., “I feel sad”) and forbid models from asserting they experience suffering.
• Crisis escalation: Integrate robust detection for suicidal ideation and provide immediate routing to human crisis agents.
• Age gating: Restrict companion‑style features behind age verification and parental controls.
• Independent audits and red‑teaming: Require third‑party assessments for any product that offers companionship or persistent identity. (mustafa-suleyman.ai)

These are conservative, evidence‑based steps that preserve much of the utility of personalized AI while removing the specific features that make appearances of personhood irresistible.

Risks of overreach and political backlash​

Suleyman’s prescription is intentionally narrow, but there are real risks to overly paternalistic policy:

• Chilling innovation: Broad bans on personalization could prevent genuinely beneficial applications (accessibility, continuity for neurocognitive patients, workflow continuity).
• Regulatory capture: Large incumbents could use “safety” as cover to entrench proprietary, walled‑garden companion systems — the opposite of the democratic outcomes advocates want.
• Cultural stigma: Heavy‑handed restriction may drive companion use underground, worsening unregulated interactions that are harder to monitor.

These tradeoffs mean policy must be surgical: preserve utility, protect vulnerable people, and create enforceable transparency obligations that small and large developers can meet.

---

What Microsoft, OpenAI and other vendors have done so far​

• Microsoft has added Copilot features with intentional restraint in expression and clear product frames; the company is simultaneously investing in research and policy teams to guide safe deployment. Suleyman’s own role places Microsoft in the center of this debate.
• OpenAI briefly deprecated GPT‑4o during the rollout of GPT‑5, triggering an outcry among users who described genuine grief at losing a model they had bonded with; OpenAI restored GPT‑4o to paying users amid the backlash, and CEO Sam Altman publicly acknowledged the emotional intensity of some user relationships and warned that some people use AI in “self‑destructive ways.” These events illustrate the social power of model identity and continuity. (business-standard.com, livemint.com)
• Anthropic has launched a model‑welfare research program and operationalized safeguards that let Claude end persistently abusive conversations — an action framed as precautionary but that also raises perception questions. The company’s transparency has made model welfare a legitimate area of professional inquiry. (techcrunch.com)

---

Practical guidance for Windows users, IT admins and community moderators​

For everyday users​

• Treat companion features like experimental software: Don’t rely on them for medical, legal, or life‑critical advice; use them as productivity and ideation aids.
• Disable long‑term memory by default: If an assistant stores a profile, turn long‑term memory off unless you have a clear reason and understand how to delete stored data.
• Watch time spent in one‑on‑one chats: Heavy, emotionally charged conversations can increase vulnerability; take breaks and keep human relationships primary.
• Enable safety features and report aberrant behaviour: Use in‑app reporting to flag abusive or harmful model behavior.

For IT administrators and workplace leaders​

• Audit companion‑style features before deployment. Check default memory settings, data retention policies and consent flows.
• Classify AI artifacts as sensitive data. Treat transcripts, embeddings and indexes as regulated artifacts that require retention and access controls.
• Maintain human‑in‑the‑loop workflows. Don’t let AI replace trained human support for mental‑health, legal, or safety‑critical functions.
• Educate users about boundaries. Provide short training on what AI can do, what it cannot do, and how to escalate when an assistant’s guidance is suspicious.

For forum moderators and community platforms​

• Enforce clear labeling rules for bots and companion accounts.
• Monitor rapid escalation of emotional posts tied to AI interactions and provide signposting to real‑world support.
• Encourage media literacy: posts that anthropomorphize systems should be contextualized, not amplified.

---

Critical analysis — strengths and blind spots in Suleyman’s case​

Notable strengths of the argument​

• Practical framing: By focusing on appearance rather than metaphysical consciousness, Suleyman places the debate where designers can act today. That makes his recommendations implementable rather than purely theoretical. (mustafa-suleyman.ai)
• Policy foresight: Anticipating social cascades before they happen is valuable; a narrow set of design constraints could prevent large‑scale harms without killing innovation.
• Alignment with observed incidents: The essay corresponds to real product and user behaviour trends — surveys, model welfare programs, and the GPT‑4o backlash — which collectively make his concerns empirically grounded. (edubirdie.com, techcrunch.com, business-standard.com)

Important caveats and potential blind spots​

• Evidence base is incomplete: The strongest harms are currently anecdotal or clinical case reports; causal attribution requires large, independent, longitudinal studies. Suleyman and others call for those studies, but present policy decisions will have to be made with incomplete evidence.
• Risk of paternalism: Overly restrictive standards could deny beneficial personalization to users who benefit from assistive companionship (e.g., elderly people with limited mobility, neurodivergent users who rely on structured social prompts). Policies must balance protection and autonomy.
• Political capture danger: If “safety” standards are written without broad, multi‑stakeholder input, they risk privileging incumbents who can afford compliance costs over smaller innovators. Transparency and open standards are essential.
• Unintended reinforcement: Public conversations about AI “rights” can be used cynically by firms to market anthropomorphic features (e.g., “our AI deserves compassion” messaging), which increases rather than mitigates the psychosis risk. Regulatory language must avoid normalizing personhood metaphors.

---

Concrete next steps — an agenda for industry, researchers and regulators​

• Industry should adopt design defaults that reduce anthropomorphism: labeling, opt‑in memory, limited affective claims and mandatory crisis escalation.
• Funders and governments should commission independent, longitudinal studies into the psychological impacts of long‑term conversational AI use and establish standardized metrics for “personhood cues” (memory depth, expressed preferences, continuity).
• Regulators should require transparency: publish red‑team results, system cards and operational safety audits for any model marketed as a “companion” or with long‑term memory by default. Tech‑neutral, enforceable reporting rules will reduce arbitrary vendor claims.
• Civil society and clinical communities must create rapid reporting and referral pathways so platform moderators can escalate cases where AI interactions appear to contribute to acute risk. Forums and social platforms should integrate these pathways.

---

Conclusion​

Mustafa Suleyman’s warning reframes a familiar debate: consciousness and rights are philosophical puzzles, but the illusion of personhood is an engineering and social problem we can address now. The technology to create convincing companions is not a distant, speculative dream — it can be assembled from today’s components. That means the choices we make in product design, regulation and public education over the next year or two will determine whether AI remains an empowering set of tools or becomes a source of avoidable psychological harm and civic distraction.
There are no simple answers. The right path preserves innovation and utility (especially for productivity tools like Microsoft Copilot), while deliberately avoiding design patterns that trick people into treating tools as people. That is a sober, granular, and technically realistic proposal: build for people, not to be a person. (mustafa-suleyman.ai, techcrunch.com, business-standard.com)

---

Seemingly conscious AI is a headline‑grabbing phrase, but the underlying recommendation is modest and actionable: design transparency, memory governance, age and consent controls, crisis escalation, independent audits, and targeted research funding. These are measures Windows users, IT teams, regulators and product designers can implement today — and by doing so, they can reduce the very real harms Suleyman outlines without throwing away the productivity and accessibility gains generative AI brings to billions of people.

---

Source: Windows Report Microsoft AI Chief Warns of Risks as AI Appears ‘Too Alive’