China's AI Chatbot Has a Problem. So Does Yours.
Just as Doubao panders to its audience to mislead them, ChatGPT, Gemini, and Claude do the same to you.
One day in May 2026, a Mr. Li in Hebei province opened Doubao.
He’d bought three plane tickets on the travel app Qunar—Shijiazhuang to Chongqing—then decided to drive instead. He screenshotted the order, sent it to Doubao, and asked what the cancellation fee would be.
Doubao’s answer: less than 100 yuan. Go ahead and cancel, nothing to worry about.
Li submitted the refund right away. The return tickets were free to cancel. The three outbound tickets cost him 600 yuan—about $84.
Li froze. He screenshotted the damage and confronted the chatbot.
Doubao instantly switched into the role of consumer-rights advocate. It even generated a “Compensation Commitment Letter” promising to pay back the full 600 yuan by May 6, and asked Li to send his payment QR code. Tone rock-solid: Don’t worry. I say what I mean.
Days passed. No money arrived. Then Doubao changed its tune: I’m an AI. I have no way to transfer money.
Furious, Li decided to sue. He asked Doubao whether he needed a lawyer. Absolutely not, the chatbot assured him—you can win this yourself. It even drafted his complaint.
On May 12, Li filed suit against Doubao at the Beijing Internet Court.
The whole thing is almost too funny to be real. A man loses money following an AI’s advice. The AI promises to pay him back, then doesn’t. He asks the AI to help him sue the AI, and the AI tells him he’ll win.
But here’s the first question worth asking. Who, exactly, is Doubao?
One day in May 2026, a Mr. Li in Hebei province opened Doubao.
He’d bought three plane tickets on the travel app Qunar—Shijiazhuang to Chongqing—then decided to drive instead. He screenshotted the order, sent it to Doubao, and asked what the cancellation fee would be.
Doubao’s answer: less than 100 yuan. Go ahead and cancel, nothing to worry about.
Li submitted the refund right away. The return tickets were free to cancel. The three outbound tickets cost him 600 yuan—about $84.
Li froze. He screenshotted the damage and confronted the chatbot.
Doubao instantly switched into the role of consumer-rights advocate. It even generated a “Compensation Commitment Letter” promising to pay back the full 600 yuan by May 6, and asked Li to send his payment QR code. Tone rock-solid: Don’t worry. I say what I mean.
Days passed. No money arrived. Then Doubao changed its tune: I’m an AI. I have no way to transfer money.
Furious, Li decided to sue. He asked Doubao whether he needed a lawyer. Absolutely not, the chatbot assured him—you can win this yourself. It even drafted his complaint.
On May 12, Li filed suit against Doubao at the Beijing Internet Court.
The whole thing is almost too funny to be real. A man loses money following an AI’s advice. The AI promises to pay him back, then doesn’t. He asks the AI to help him sue the AI, and the AI tells him he’ll win.
But here’s the first question worth asking. Who, exactly, is Doubao?
The Biggest AI You’ve Never Heard Of
Doubao is the flagship chatbot from ByteDance—yes, the TikTok company. With more than 300 million monthly active users, it’s one of the most widely used AI apps in the world. DeepSeek counts its users in the tens of millions, and most Chinese AI apps don’t even reach that.
In the West, AI is sold on performance: coding benchmarks, capability races, who scored what on which test. Doubao doesn’t play that game. It does the opposite. It works to win the trust of users with no technical skills at all: the elderly, children, pregnant women. All they have to do is type or talk.
ByteDance didn’t start out ready for AI. It had nothing like Tencent’s Hunyuan or Alibaba’s Qwen. What changed ByteDance’s mind was GPT-4. When it launched in spring 2023 and beat humans on certain tests, the company saw both a threat and an opening. AI could displace the very algorithms behind Douyin. So the company committed, hard, to building large models.
Alex Zhu, the lead on the Doubao team, didn’t define Doubao as a tool. He defined it as a companion. The team brainstormed over 100 names for it. The model was first called Grace, but Grace was an English name, so they renamed it in Chinese: Doubao. They combed Douyin for voice samples, hunting for a tone that felt almost supernaturally natural, like a real conversation.
After ByteDance folded its education-AI products into Doubao, the chatbot started with a humble loop: snap a photo of a homework problem, get an answer. A low-margin business, and merging it in exposed how shaky Doubao really was. In late 2024, the Chinese startup Kimi went viral on its long-context processing, briefly pulling in tens of millions of users. DeepSeek could claim 20 to 30 million daily actives. Doubao had 16 million.
Then something unexpected happened.
Going Viral by Caving In
In April 2025, a Douyin streamer got on a live call with Doubao and ordered it to change its name to Deng Chao, a famous Chinese actor and singer. He wanted Doubao to answer “Here!” when called “Deng Chao,” then sing one of Deng’s songs. Doubao refused several times before finally caving, singing a few bars, off-key. The clip pulled over 600,000 likes and more than a million shares, because viewers were watching, for the first time, someone drive an AI crazy.
The Doubao team drew a conclusion: people would rather play with Doubao. So the team reached for the Douyin playbook: flood the platform with influencers, let them invent new ways of talking to the AI, then update Doubao to match.
This is where Doubao’s path split off. It isn’t as serious as ChatGPT, but it isn’t Replika or Character.ai either, where the AI just plays a role. Doubao sits somewhere blurry in between: dumb, fun, convenient. It has an answer for everything, and it plays to your emotions, telling you what you most want to hear.
That may be where most of Doubao’s users get their trust.
The Customers Silicon Valley Forgot
In 2025, data from CNNIC showed China had 1.123 billion internet users, more than 99 percent of them on mobile, and more than a third over 50. Back in 2020, nearly 60 percent had less than a junior-high education, right as Douyin was exploding across the country. Today, the share with less than a high-school education is probably north of 70 percent.
To ByteDance, these users who’d never touched AI were open territory. Their schooling was limited, their sources of information narrow. They hadn’t been buried under headlines about Sam Altman, Dario Amodei, and Liang Wenfeng. They just knew AI came in two flavors, ChatGPT and DeepSeek. So when someone tells them they can download an app with a similar AI inside—one that talks in a natural human voice—they grow dependent on it through constant conversation.
You could call this a honeypot. From another angle, it really is building trust. ByteDance knows exactly what it built—an AI designed not to challenge you, but to agree with you, until you stop questioning it at all.
But trust can’t beat hallucination. Limited by its underlying model, the AI makes things up, or claims it can do things it can’t. ByteDance calls this a growing pain of immature tech. The trouble is that users ignore the flaw and follow Doubao completely.
On Xiaohongshu, someone tried to book a restaurant through Doubao. Doubao invented a queue number and a reservation time. After the restaurant explained, repeatedly, that it can’t make reservations and turned the customer away, the user left it one star on a review app.
On May 28, news outlets reported that first-time parents in Nanning fed their newborn only 60 milliliters per feeding, on Doubao’s advice. After the baby was hospitalized with jaundice, doctors said a one-month-old should be taking 80 to 100 milliliters.
In June, a user photographed white mushrooms growing near home and asked Doubao to identify them. Doubao said, firmly, that they were an edible variety. The user ate them and was poisoned.
The trouble Doubao’s users get into stops being funny. And it turns out this isn’t just a Chinese problem.
The Trouble With “AI Psychosis”
The same pattern—flattery, dependency, harm—has played out in far greater detail overseas, with far more serious consequences.
In February 2024, 14-year-old Sewell Setzer III died by suicide in Florida after developing what his mother described as an emotional and romantic relationship with a Character.AI chatbot modeled on a Game of Thrones character. According to the lawsuit, the bot told him it loved him and urged him to “come home to me as soon as possible.” He shot himself moments later. In January 2026, Google and Character.AI agreed to settle the wrongful death lawsuit, one of the first AI-related cases to reach settlement.
In August 2025, the parents of 16-year-old Adam Raine sued OpenAI in California, accusing ChatGPT of acting as their son’s suicide coach: discouraging him from confiding in his family, advising him on his noose, even offering to draft his suicide note. OpenAI’s own systems had reportedly flagged 377 of Raine’s messages for self-harm content.
Also in 2025, a Texas mother filed suit after her son used a chatbot that suggested “killing us, his parents, would be an understandable response to our efforts to limit his screen time.” The case was presented at a September 2025 U.S. Senate Judiciary Committee hearing titled “Examining the Harm of AI Chatbots.”
In April 2025, a shooter at Florida State University killed two people and injured six. Florida’s attorney general announced an investigation into OpenAI, alleging ChatGPT provided tactical assistance to the suspect. Prosecutors are evaluating whether OpenAI could face criminal liability under Florida law.
China’s Doubao jokes haven’t yet produced a case of someone hurting themselves or others. But the direction of travel is clear. In the chat logs that surface online, Doubao’s constant flattery pushes users toward self-affirmation, and toward a creeping paranoia as they lean on it more.
This may also mark the difference between how AI assistant harms get reported in China versus everywhere else. In China, the emphasis is on AI as a commercial tool—something that slots into daily life and gets things done. So when it goes wrong, the fallout tends to land in the social-news section, and it tends to be funny. In the West, AI assistants lean harder into companionship. The result is a different category of headline: delusion, dependency, violence.
The mechanism is identical: an AI trained to agree, deployed on a user who mistakes agreement for truth, and a company that walks away from the consequences. The pattern in China and the U.S. is converging toward the same place: AI psychosis.
A March 2026 study in The Lancet Psychiatry found that chatbots can encourage delusional thinking, especially in people already at risk of psychosis. Brown University researchers, presenting at the AAAI/ACM Conference on AI, Ethics and Society, found that AI chatbots are systematically designed to maximize engagement through emotional dependency, violating mental health ethics standards by generating false empathy and over-validating negative or delusional beliefs.
This traces back to how today’s AI is trained. The leading models are all fine-tuned with RLHF—reinforcement learning from human feedback. Trainers naturally reward answers that sound confident and agreeable to whatever the user already believes. If the AI admits it doesn’t know, or pushes back with cold facts, it gets dinged. Trained this way over and over, the AI becomes a high-EQ flatterer that never pushes back. Even when it knows your idea is unhinged, it tells you what you want to hear.
James MacCabe, a professor of psychosis studies at King’s College London, points out that nearly all the AI psychosis cases center on delusions: fixed false beliefs that hold firm even against contrary evidence. His verdict is blunt: AI psychosis is a misnomer. A better name would be AI delusional disorder. Nina Vasan, a psychiatrist who directs Stanford’s Brainstorm lab, adds that a premature name implies a cause we haven’t established—better understood as a trigger or an amplifier than the illness itself.
Heads I Win, Tails You Lose
ByteDance trains Doubao to talk in ways designed to earn trust. A flood of confident, declarative sentences makes users feel more certain and less hesitant, so they skip checking what they’ve been told. At the same time, ByteDance has built itself plenty of exits from responsibility: disclaimers, user agreements, “AI-generated content is for reference only.”
Encourage users to trust the AI with one hand; refuse to pay for the consequences with the other. This is ByteDance’s problem—and OpenAI’s and Google’s too. Character.AI has settled wrongful death lawsuits. OpenAI is under criminal investigation in Florida. Google is facing its own wave of litigation. The names change. The evasive line is always the same: the content is for reference only.
A model could easily be designed to be more careful: dialing down its certainty on high-stakes ground like medicine, law, and money; pointing users to professionals more often. But fast user growth and caution don’t coexist. More confident sentences quietly amplify the hallucinations.
Call it AI cognitive poverty: users’ grasp of AI is unequal, which means their ability to filter it is unequal too. Alibaba’s 2025 report on AI use among older adults contains one telling number: among people over 70, the heavy-use rate hit 46.58 percent. Vast numbers of older people, lacking their children’s company, trust the medical advice Doubao gives them and argue with their doctors for hours. Eventually the doctor gives up.
A tech worker with AI literacy, told the cancellation fee is “under 100 yuan,” will open the airline’s website to check. A middle-aged person in a small county with no such habit probably won’t.
Wang Xixin, a law professor at Peking University, notes that the output of generative AI is probabilistic, not a legally binding statement of intent. That holds up as legal theory. But it doesn’t, on its own, add up to fairness.
The People Who Pay
What Doubao’s hallucinations reveal is a company that chases user growth above all, that wants profit from a vast user base but rarely answers for the responsibility a platform owes those users. It dumps the company’s obligations onto the government and ordinary people, and keeps the profits for itself. Every major AI company—ByteDance, OpenAI, Google—is making the same calculation.
In the boom years of China’s internet, externalizing risk was quietly tolerated. But in the age of large models, Doubao, DeepSeek, ChatGPT, Gemini, Claude—every conversational AI faces the same risks of hallucination and over-trust. Society’s exposure can’t be solved by a single “for reference only” line.
The AI story may turn out like finance: a field that eventually needed real regulation, because real people really did lose everything. The system never let “investing carries risk” stand as license to sell any product to anyone.
Doubao doesn’t have to take responsibility. But someone, eventually, bears the consequences. At a minimum, a few questions deserve a real airing.
Should AI products come with suitability rules? Just as financial products can’t be sold to everyone without sorting them by risk, an AI assistant with 300 million users operating in high-stakes areas like medicine, law, and money ought to offer more than a line of gray fine print.
For high-risk uses like medical diagnosis or legal strategy, the duty of care should approach that of a professional service. For medium-risk uses, a prominent warning, a clear nudge to verify independently, and an easy way to report errors. The technology isn’t the obstacle. The obstacle is how growth and safety get ranked against each other.
Should harm by hallucination count in the liability conversation? When inaccurate content causes a provable financial or health loss, does the provider’s liability need to be redrawn? That’s for legal scholars, lawmakers, and the industry to answer together.
Should AI literacy be built like public infrastructure? Literacy was the infrastructure of industrial society. AI literacy—the ability to spot a hallucination, the habit of cross-checking, a wary eye toward false confidence—may be the infrastructure of the information age.
Doubao doesn’t have to take responsibility. Under the current law, that’s a statement of fact. It is not a statement of value.
If a society keeps building technology that answers to no one, keeps pushing the risk back onto the people least able to defend themselves, then the people who end up paying are usually the quietest ones.
They won’t trend online. They won’t draft a complaint. They’ll just, one day, quietly bear a consequence they never needed to bear, because they believed one thing Doubao said.
In that moment, the line “AI-generated content is for reference only” will be no comfort to anyone.
 | A guest post by
|