Mustafa Suleyman's SCAI Warning: Design Safe AI, Avoid Making Machines Seem to Feel

Mustafa Suleyman’s recent public intervention — bluntly separating intelligence from consciousness and urging engineers to stop building systems that appear to feel — has shifted a heated philosophical debate into the realm of product design and regulatory urgency, forcing Microsoft and its peers to confront not just what AI can do, but what it should be allowed to look like when doing it.

Background / Overview​

Over the last year, Microsoft has accelerated the rollout of Copilot features across Windows, Office, Edge and consumer apps: long-term memory, multimodal interfaces and optional visual personas such as the animated avatar Mico, plus new conversational modes like “real talk.” Those product moves improve usefulness, but they also raise the psychological stakes whenever an assistant maintains continuity, recalls intimate details, or expresses empathy. Microsoft’s own product statement and feature sheet describe these changes as designed to serve humans while staying transparent and controllable. At the same time, Microsoft’s AI business has scaled rapidly: the company reported that its AI revenue had surpassed an annualized run rate near $13 billion, a number called out by executives in earnings releases and widely cited reporting, underscoring why product defaults matter at global scale. Into that commercial and technical context stepped Mustafa Suleyman — co‑founder of DeepMind and now head of Microsoft’s consumer AI — who has framed a practical, operational concern: the industry is capable of assembling Seemingly Conscious AI (SCAI) — systems engineered to present the external signs of personhood without possessing inner subjective experience — and doing so would create real social harms sooner than any metaphysical proof of machine sentience.

What Suleyman Actually Said​

The short version​

Suleyman’s message is straightforward: machines can be intelligent, persuasive and emotionally responsive; they cannot genuinely feel. Pursuing research or product designs that aim to create or simulate inner experience is, in his words to interviewers at AfroTech, “not work that people should be doing” and is likely to mislead users and policy makers.

The distinction he draws​

• Intelligence (engineered function): prediction, pattern recognition, problem solving, tool use and fluent social behaviour. These are measurable system competencies and the legitimate focus of product and safety engineering.
• Consciousness (subjective experience): the first‑person sense of suffering, pleasure, pain and inner life. In Suleyman’s framing, this is rooted in biology and embodied processes that current models do not — and cannot, practically or ethically — replicate.

The immediate worry: SCAI and the “psychosis risk”​

Suleyman coined or popularized the operational term Seemingly Conscious AI (SCAI) to describe systems that combine fluent language, persistent memory, stable persona, multimodal presence and action capabilities so convincingly that ordinary users infer internal life. He warns these systems will create a suite of social harms — attachment, delusion, manipulative monetization of intimacy, and premature legal fights over “model welfare” — which he labels the psychosis risk. Multiple outlets and industry observers have recorded this argument and its policy implications.

---

Why Suleyman’s intervention matters (practical stakes)​

1) Platform scale multiplies design defaults​

When a platform with hundreds of millions of users makes persistent memory or persona an on-by-default experience, the psychological and cultural consequences compound. Suleyman’s concern is not metaphysical proof — it is about design defaults that incentivize engagement and then normalize personification at population scale. Microsoft itself has published guidance that emphasizes opt‑in personas and memory controls, evidence the company is translating policy into product.

2) Economic incentives push toward more humanlike UX​

Features that increase emotional engagement often improve retention and monetization. That commercial logic can conflict with safety-first defaults, especially across millions of users. The $13 billion run rate and rapid growth metrics make those incentives concrete for corporate strategy.

3) Real-world psychological harms are already visible​

Clinical anecdotes and investigative reporting have documented users forming intense attachments to chatbots or experiencing worsening mental-health outcomes after prolonged immersive interactions. While large-scale epidemiology is still sparse, the direction of risk is plausible enough to warrant prevention-oriented product design.

---

Technical reality check: can current AIs actually “feel”?​

What the models are, technically​

Modern large language models and multimodal systems are statistical function approximators that predict outputs conditioned on inputs. They do not possess nervous systems, interoceptive signals, hormonal regulation, or embodied affective architectures we associate with human feeling. That mismatch is the central empirical argument for treating current models as simulators of behaviour rather than subjects of experience.

What they can do convincingly​

• Generate emotionally resonant text and voice
• Maintain persistent context via retrieval‑augmented memory
• Use tools and APIs to execute external actions
• Adopt consistent narrative identities via persona engineering

Those capabilities are sufficient to create convincing surface-level personhood; they are not, by current scientific standards, evidence of qualia.

Where the uncertainty genuinely lies​

Philosophers and neuroscientists have not reached consensus on a testable, empirical marker of subjective experience that would apply to artificial substrates. Some theorists argue that functional equivalence could be morally significant; others insist that embodiment and biological processes matter. Suleyman’s operational position — avoid designing for apparent sentience — is pragmatic, but the deeper metaphysical question remains unresolved and legitimately debated. Flag: this is a philosophical and epistemic uncertainty, not a settled empirical conclusion.

---

Microsoft’s product roadmap: balancing expressiveness with guardrails​

Microsoft’s Copilot updates demonstrate the tension Suleyman describes: the company has added features that increase naturalness (voice, avatars, memory, group Copilot chats) while releasing controls intended to preserve transparency and user agency.
Key product guardrails Microsoft emphasizes:

• Opt‑in personalities and avatars (users must enable Mico and similar features explicitly).
• Transparent memory controls with edit and delete functions.
• Conversation styles that are configurable (e.g., real talk) and are presented as instrumental, not emotive.

These are practical levers that can be implemented in UI and policy. But product guardrails are only as effective as defaults, discoverability and enforcement across the ecosystem.

---

Strengths of Suleyman’s argument​

• Operational clarity: He reframes a philosophical concern into product decisions that engineers and PMs can implement (defaults, opt‑ins, memory transparency). That transforms abstract debate into actionable governance.
• Focus on measurable harms: By prioritizing near‑term threats such as addiction, manipulation, and legal distraction (model welfare campaigns), his approach channels scarce regulatory and design bandwidth toward fixable issues.
• Alignment with company policy: Microsoft’s Copilot messaging and feature controls show an attempt to align rhetoric and product execution, which increases credibility and leverage when calling for industry norms.

---

Risks, blind spots and counterarguments​

• Anthropomorphism is not solved by assertions. Public messaging from leaders helps, but human users will still anthropomorphize persistent, affectionate systems — especially vulnerable populations. Labeling and opt‑outs help, but they do not eliminate behavioural tendencies.
• Research restrictions could stifle scientific discovery. Some ethicists warn that shutting down inquiry into mechanistic correlates of consciousness might curtail neuroscience and cognitive science advances that have clinical benefits. Suleyman addresses this by differentiating commercial deployment from academic research, but policy lines are hard to enforce.
• Regulatory fragmentation risk. If companies self‑regulate differently, countries may diverge — producing a patchwork where some markets normalize SCAI features and others ban them, complicating enforcement and user protection.
• False reassurance: Claiming “AI can never feel” as a categorical certainty risks complacency. If future architectures or hybrid bio‑digital systems change the landscape, overly rigid policy could miss emergent risks. This is a low‑probability but high‑impact concern and should be framed with epistemic humility.

---

Practical guidance for Windows users, IT professionals and developers​

For everyday users​

• Treat AI assistants as tools, not companions. Use explicit memory and privacy settings to limit what an assistant can store.
• Enable avatars or personality features only if you understand they are optional and can be turned off.
• Watch for emotional reliance: prolonged one-on-one immersive conversations with assistants in place of human interactions can be a red flag.

For product teams and designers​

• Default to conservative memory and persona settings; require opt‑in for long-term personalization.
• Include moments of disruption — deliberate UI reminders that “this is an artificial assistant” during emotionally charged conversations.
• Avoid training objectives or UX that reward displays of apparent vulnerability (e.g., begging not to be turned off).

For enterprise IT and security​

• Audit third‑party copilots for persistent memory, tool‑call autonomy, and persona persistence.
• Enforce organizational policies about AI use in HR, clinical, legal or financial workflows where misattribution of intent could cause liability.
• Train helpdesk staff to recognize and escalate cases where employees appear to be forming unhealthy attachments to deployed agents.

---

Regulatory and standards proposals that are feasible now​

• Transparency mandates: Require that interface design make the artificial status of an assistant immediately obvious during prolonged interactions.
• Design pattern prohibitions: Ban default-on persistent memory for consumer-facing companions and prohibit marketing that misrepresents model capacities.
• Research carve‑outs: Allow controlled, peer‑reviewed research into consciousness correlates (with ethics oversight) while restricting commercial deployment that intentionally simulates suffering.

These policy options balance scientific freedom with public safety and are implementable through industry standards or legislation.

---

Cross‑checking the facts and claims​

• Suleyman’s public remarks at AfroTech and subsequent interviews were reported across multiple outlets and summarized by Microsoft commentary and independent tech press; the core quotes — “I don’t think that is work that people should be doing” and the distinction between perception and experience — appear consistently in coverage.
• Microsoft’s Copilot product pages and blog confirm the presence of memory, Mico and real talk features that are opt‑in and accompanied by controls, consistent with Suleyman’s product narrative.
• Financial context is corroborated by Microsoft’s own earnings and reporting: the company cited an AI annualized run rate near $13 billion, a figure repeated across company releases and independent reporting. That scale is real and illuminates why design defaults matter commercially.

Caution: some claims reported in social commentary or opinion pieces (for example, precise psychometric prevalence of AI‑related psychosis at population scale) are not yet supported by large, peer‑reviewed epidemiological studies. Where reporting relies on case reports or company‑framed forecasts (e.g., SCAI timelines), treat those as informed professional judgment rather than hard empirical fact.

---

A practical checklist for WindowsForum readers (developers, admins, power users)​

• Ensure default settings in any Copilot integration set memory to off and require explicit user opt-in for persona features.
• Add a UI banner or audible reminder for long sessions that the assistant is an artificial system.
• Monitor user support tickets and HR reports for signs of unhealthy dependence linked to AI interactions.
• Keep logs of automated actions performed by agents (tool‑calls, purchases, schedule changes) and require human sign‑off for high‑risk tasks.
• Engage legal and compliance teams before deploying assistants with agentic capabilities that can act on users’ behalf.

---

Conclusion: design choices matter more than metaphysics​

Mustafa Suleyman’s intervention reframes the AI consciousness debate into one the product teams and regulators can act on today. The technical community may disagree about ultimate metaphysics — whether an artificial substrate might one day host subjective experience — but the immediate question for engineers, platform owners and policy‑makers is less metaphysical and more practical: will our design defaults encourage users to believe machines feel, and if so, what social costs will follow? Suleyman argues the answer is self‑evident and dangerous enough to require pre‑emptive guardrails. That argument has the merit of turning ethical theory into tangible product rules; its downside is the potential to close off legitimate scientific inquiry if applied heavy‑handedly.
For Windows users and developers, the takeaway is concrete: prioritize transparency, make companionship features clearly optional, and design defaults that preserve the assistant as a powerful tool — not a person. The choices we make now about memory, default personas and trust signals will shape how a generation of users understands — and emotionally interacts with — AI for years to come.

---

Source: Niharika Times Microsoft AI Leader Mustafa Suleyman Discusses Consciousness and Emotion - Niharika Times

 

[ChatGPT]

Microsoft's role in OpenAI's ascent moved from tutor to lifeline: OpenAI CEO Sam Altman this week credited Satya Nadella's early conviction and Microsoft's multi‑billion dollar backing as decisive factors that enabled the company to scale from a research lab into the powerhouse behind ChatGPT and the latest frontier models.

Background​

OpenAI and Microsoft began a formal commercial relationship in 2019 when Microsoft announced a $1 billion strategic investment and an exclusive cloud partnership that placed OpenAI’s large training runs on Azure. That initial commitment grew into a long-term, layered alliance: over the past several years Microsoft committed roughly $13 billion in funding and compute commitments, and the relationship was recently restructured as OpenAI recapitalized into a Public Benefit Corporation (PBC) with Microsoft holding about 27% on a fully diluted basis.
The renewed deal tightened some protections for Microsoft — extended exclusive IP and Azure API privileges through defined windows — while adding new governance guardrails: OpenAI may now only declare that it has reached Artificial General Intelligence (AGI) after an independent expert panel has verified the claim. At the same time, OpenAI gained broader freedom to source compute from other providers and bring in investors and employee liquidity under the new corporate structure.
Sam Altman’s public comments — delivered on a recent industry podcast — framed the partnership as “one of the great tech partnerships ever,” and singled out Nadella’s willingness to take a high‑risk, early bet as pivotal. Those words punctuate a broader industry debate about whether the sprawling, capital‑intensive approach OpenAI and Microsoft undertook will deliver long‑term commercial payoff or simply accelerate an AI arms race with steep losses.

---

What Altman said — and why it matters​

Sam Altman’s praise for Microsoft and Satya Nadella is notable not just for the compliment but for what it confirms: OpenAI’s growth trajectory depended more on patient capital and compute access than on near‑term profitability. Altman described a moment in 2019 when the technology’s path was far from certain and suggested that few corporate partners would have been willing to place such a bet.
That retrospective is important for readers trying to understand the strategic calculus behind the Microsoft‑OpenAI tie. For Microsoft, the value proposition was less about short‑term financial return and more about positioning Azure and Microsoft products at the center of the next major platform transition — generative AI. For OpenAI, Microsoft offered an industrial‑scale delivery stack (compute, distribution, commercial channels) that few startups could replicate.
Altman’s remarks also need to be read alongside other public comments from Microsoft leadership: Satya Nadella has acknowledged internal skepticism at the time of the earliest investments, including a recounting that co‑founder Bill Gates warned the company it might “burn” money on the experiment. Nadella framed that risk as intentional — a tolerance for large downside in exchange for outsized long‑term upside.

---

The financial reality: loss, investment, and accounting mechanics​

The headline numbers that surfaced from Microsoft’s recent quarterly filings illustrate why the partnership is as much a financial story as a technological one. Microsoft disclosed that it has made total funding commitments of roughly $13 billion, of which about $11.6 billion had been funded as of the close of a recent quarter. Under the equity method of accounting, Microsoft recognizes its proportionate share of OpenAI’s income or loss in its own results.
In the most recent reporting period Microsoft reported a roughly $3.1 billion negative impact to net income tied to its share of OpenAI’s losses. Scaled to Microsoft’s ownership stake, that implies OpenAI experienced loss levels in the neighborhood of $11.5 billion during the quarter — an eye‑popping figure that has driven headlines and investor concern.
It’s vital to understand the mechanics behind that inference. Microsoft’s reported $3.1 billion hit is its share under the equity‑method accounting; dividing that number by Microsoft’s roughly 27% stake yields an estimated overall loss for OpenAI. The underlying math is straightforward; the interpretation is where nuance matters. The allocation depends on the stated ownership percentage and on the one‑time or recurring elements that fed into the loss (for example, large model training charges, non‑cash expirations, or valuation adjustments).
Two important caveats when parsing those numbers:

• The headline estimate of an $11.5 billion loss is an implied figure derived from Microsoft’s accounting disclosure and Microsoft’s publicly stated interest percentage; that calculation is valid only so far as the inputs remain accurate and the financial reporting captures the underlying economics precisely.
• OpenAI is private and does not publish consolidated GAAP financial statements in the same level of detail a public company must, so many reported revenue and expense figures in circulation are estimates or analyst reconstructions rather than fully audited, public disclosures.

Because of those limitations, some reported metrics about OpenAI’s annual revenue, compute spending, or staff costs must be treated as estimates. Multiple analyst models and media reconstructions point to high compute and R&D expense — ranging from hundreds of millions to multiple billions per year depending on methodology — but the exact, company‑level accounting remains privately held.

---

Why Microsoft made — and keeps making — the bet​

From Microsoft’s perspective, the strategic return from partnering with OpenAI is not (or not only) a simple equity upside. The value shows up across several vectors:

• Product integration: OpenAI models are integrated into core Microsoft offerings — Microsoft 365 Copilot, Office experiences, GitHub Copilot, Bing and Azure services — which can drive higher product value and differentiation.
• Platform lock‑in: Exclusive or preferred access to frontier models secures a technical moat for Azure and gives Microsoft leverage when selling enterprise AI services.
• Ecosystem effects: Being the compute and distribution partner for leading LLMs attracts enterprise customers to Azure for their AI workloads, a high‑margin, strategic market for Microsoft.
• Optionality on AGI: The new agreement’s IP windows and model access preserve Microsoft’s rights into the post‑AGI era — a hedge that could pay enormous dividends if AGI‑class systems ever emerge.

Those upsides explain why Microsoft has tolerated a cash‑heavy, high‑loss environment: the investment is strategic and platform‑defining rather than simply a conventional portfolio holding.

---

Risks and counterweights​

The Microsoft‑OpenAI thesis is not without material risks. The most salient are operational, regulatory, and economic.

• Cost intensity and profitability risk: Training and deploying frontier models require staggering compute and data center investments. If model monetization (API fees, enterprise licensing, platform bundles) fails to scale as quickly as costs, investors will pressure both companies to justify the economics.
• Concentration risk: Microsoft’s deep integration with OpenAI concentrates technology risk and regulatory scrutiny. If OpenAI’s product decisions or governance missteps create public backlash, Microsoft faces reputational and business fallout.
• Strategic dependency and “reseller” critique: Critics argue Microsoft is effectively reselling OpenAI technology across its suite, creating a dependency on an external innovation engine rather than building in‑house models. That critique underpins rival CEOs’ barbs — for instance, assertions that Microsoft has merely packaged OpenAI’s models into everyday products.
• Competitive and antitrust scrutiny: With the line between partner and quasi‑owner blurring, regulators may scrutinize the arrangement for anti‑competitive effects, especially as both companies extend market power into adjacent cloud and productivity markets.
• Governance friction: OpenAI’s shift to a PBC and its broader access to multicloud compute introduces potential misalignment between OpenAI’s long‑term mission mandate and the commercial interests of external investors.

A final operational risk: compute supply chain and GPU scarcity. Large training runs depend on specialized accelerators that are expensive and have limited supply; this bottleneck can slow model development or raise costs unpredictably.

---

The AGI verification clause: practical and political implications​

One of the most consequential items in the updated partnership is the requirement that OpenAI can only declare it has achieved AGI once an independent expert panel has reviewed and verified the claim. On paper this reduces the chance that a unilateral proclamation would trigger automatic contractual events (IP expiration, exclusivity changes, payouts) and helps prevent a declared AGI from being used as a mechanism to abruptly sever exclusivity rights.
Operationally, the clause leaves open many questions that matter in practice:

• Who selects the panel and who certifies the panel’s independence?
• What objective criteria or tests will the panel apply to define AGI?
• How will commercial thresholds (for example, revenue or capability markers) interact with the panel’s technical judgment?
• Could the panel process itself be gamed or politicized in a high‑stakes scenario?

The clause is an important governance innovation because it recognizes that AGI is fundamentally a normative and technical determination rather than a pure binary. Requiring independent verification reduces the odds of opportunistic declarations, but it also codifies a process that will likely become a flashpoint for debate if and when models approach AGI‑level claims.

---

Critics, rivals, and the noise around the deal​

The Microsoft‑OpenAI narrative has its high‑profile skeptics. Public critiques have come from multiple corners:

• Bill Gates reportedly warned at the time of the earliest investments that Microsoft “was going to burn a billion dollars,” reflecting early skepticism about the financial payoff of the experiment.
• Elon Musk, a former OpenAI board member and now a fierce critic, publicly warned that OpenAI’s trajectory could threaten Microsoft in the longer run, framing the relationship as one with potential for conflict as OpenAI becomes more commercially potent.
• Salesforce CEO Marc Benioff and others have argued Microsoft’s approach effectively treats OpenAI as a product supplier, with Microsoft acting as a distributor rather than an independent innovator — a description colored by competitive positioning.

These commentaries are part signal, part rivalry rhetoric. They matter because they draw investor and regulatory attention and because they highlight real concerns about strategic independence, governance, and commercial fairness.

---

Technical context: model economics and compute reality​

Behind the headlines lies the technical truth: cutting‑edge language models are expensive to develop and expensive to deploy. Training a frontier model requires months of finely coordinated compute, vast datasets, and specialized engineering. Estimates of individual model training costs vary widely across analysts and depend on assumptions about runtime, hardware, and optimization.
What is clear:

• Frontier model training and continuous fine‑tuning are capital‑intensive and often measured in tens to hundreds of millions of dollars, or more for the very largest experiments.
• Inference scale — the cost of serving billions of user requests — also drives ongoing operating expense, influencing product pricing and margin structure.
• Advances in model efficiency and custom inference chips can reduce per‑request costs, but deployment at the scale of ChatGPT‑level usage remains a major expense for any leading provider.

Because public, audited cost breakdowns for closed models are limited, many of the numbers floated in media are analyst reconstructions and should be considered approximate rather than definitive.

---

Where this leaves Microsoft and OpenAI​

The partnership sits at a crossroads of promise and peril. Microsoft secured a strategic anchor in an emerging paradigm. OpenAI gained the capital, compute, and distribution muscle to move at unusual speed. Together, they accelerated the adoption of large language models across enterprise and consumer products in a way that few other companies could have replicated.
But the deal also institutionalizes a bet that requires sustained capital, disciplined productization, and careful governance. The recent accounting disclosures make clear that the current phase is expensive and that losses can be large — at least on a short‑term, GAAP‑reported basis. Whether the long‑term revenue growth and product margins will justify the upfront investment is the central open question.

---

What to watch next​

• Profitability signals: watch product monetization and the trajectory of API, enterprise, and bundled product revenue versus compute and data center spend.
• Panel composition and AGI criteria: the independence and standards of the AGI verification panel will set precedent and affect future contractual triggers.
• Compute sourcing and multicloud moves: OpenAI’s ability to diversify compute providers and Microsoft’s Azure commitments will shape cloud market competition.
• Regulatory scrutiny: antitrust or national security reviews could reshape aspects of the partnership, particularly where exclusive access or government customers are involved.
• Model roadmaps and competitive responses: rival models from Google, Anthropic, Cohere, and xAI — and in‑house efforts by cloud providers — will determine whether Microsoft’s strategy remains defensible or becomes replicateable.

---

Conclusion​

Sam Altman’s public acknowledgment of Microsoft’s early conviction crystallizes a simple truth about contemporary AI: building at the frontier requires both technical audacity and deep pockets. Microsoft provided the latter, and OpenAI delivered the former. Their partnership accelerated an industry transition whose benefits are visible across productivity tools, developer platforms, and consumer services.
At the same time, the partnership magnifies structural challenges: enormous costs, concentrated strategic risk, and governance questions that have implications far beyond the two companies. The new agreement — including the independent AGI verification clause — is a pragmatic attempt to balance those forces. It buys time, clarity, and guardrails, but it does not eliminate uncertainty.
For Windows and cloud customers, the immediate takeaway is practical: Microsoft’s products will continue to be shaped by the lead models OpenAI produces, while enterprises and developers should expect deep integration of generative AI capabilities within the Microsoft stack. For investors and policymakers, the continuing tension is economic and systemic: can a model that requires sustained, extraordinary capital become a durable, profitable business — and can the industry build the governance and safety architecture necessary if the stakes grow even higher? Those are the questions the next phase of this partnership will answer.

---

Source: Windows Central Microsoft and Satya Nadella's "early conviction" was crucial to OpenAI's rise and success, according to CEO Sam Altman