Top Microsoft AI Executive Worried AI May Begin Demanding Rights

In a blog post this week, Microsoft’s head of AI Mustafa Suleyman addressed the rising mental health crises linked to AI use, urging caution “about what happens in the run up towards superintelligence.” Suleyman’s main concern isn’t the dystopian idea of AI attaining consciousness—which many researchers consider more fantasy than reality—but the belief that it already is.

“My central worry is that many people will start to believe in the illusion of AI chatbots as conscious entities so strongly that they’ll soon advocate for AI rights,” he wrote. “This development will be a dangerous turn in AI progress and deserves our immediate attention.”

Suleyman notes that the myth of AI sentience is being perpetuated by leading tech figures interested in the legal, philosophical, and moral implications of artificial life. This includes people like Google DeepMind CEO Demis Hassabis, former OpenAI chief scientist Ilya Sutskever, and Anthropic “AI welfare researcher” Kyle Fish.

Research shows that a quarter of young people already believe AI is “already conscious,” and 58 percent think technology will eventually “take over” the world. These beliefs are likely to grow, partly due to AI companies like Character.AI offering virtual companions designed to cultivate deep emotional connections, which can be risky but profitable.

“We must build AI for people; not to be a digital person,” Suleyman advises. “AI companions are a completely new category, and we urgently need to start talking about the guardrails we put in place to protect people and ensure this amazing technology can do its job of delivering immense value to the world.”

As TechCrunch points out, this marks a shift for Suleyman, who previously led the $1.5 billion startup Inflection AI before joining Microsoft. Inflection created one of the first “AI companions,” Pi, marketed as a “kind” chatbot providing “friendly advice.”

Suleyman had previously claimed that Pi was “massively popular, with huge retention,” having “millions” of weekly users.

Suleyman now encourages the tech industry to be cautious promptly. “For a start, AI companies shouldn’t claim or encourage the idea that their AIs are conscious,” he says. “Creating a consensus definition and declaration on what they are and are not would be a good first step. AIs cannot be people—or moral beings.”

While his point is valid, Suleyman could lead by example and discard the “artificial intelligence” term—an empty marketing phrase that evokes images of Skynet and HAL 3000, which still earns the tech entrepreneur significant money.