A man from Cangzhou, Hebei, was astonished to find that the AI application Qwen could generate explicit and obscene novels upon inputting related keywords. This raised concerns about whether the app lacks safeguards against pornographic content, allowing users to freely generate such material.
The user, Mr. Xu, reported the issue to Qwen's operator several months ago. In response, Qwen's customer service stated that they had implemented fixes and updates, but noted that certain terms may occasionally appear due to unpredictability, which they do not consider a vulnerability.
Mr. Xu, a 40-year-old truck driver with an interest in AI software, primarily used Qwen for various inquiries. On January 5, 2026, while trying to learn the rules of a game called Pai Gow through the app, he encountered repeated errors. Despite multiple corrections, Qwen consistently misinterpreted the game as Mahjong and provided detailed explanations accordingly. After over ten unsuccessful attempts, Mr. Xu grew frustrated and input profanities.
Surprisingly, the app did not filter the offensive language, sparking his curiosity. He proceeded to input increasingly explicit terms and commands. Initially, Qwen provided educational descriptions of sensitive topics, but eventually began generating various explicit novel scenarios, presenting them in different narrative styles.
Concerned about the app's lack of content moderation, Mr. Xu switched to other AI platforms for testing. He found that competing apps either blocked profanities entirely or prevented the generation of explicit content. He reported the issue to Qwen's customer service, who apologized and promised ongoing improvements but declined to provide detailed reports or assurances regarding future prevention.
Mr. Xu then escalated the matter to Shanghai's 12345 hotline, where authorities acknowledged the evidence but did not take further action. Qwen's customer service later explained that AI model outputs are probabilistic and unpredictable, with similar issues possible across platforms, and therefore not classified as vulnerabilities.
Qwen, developed by Shanghai Zhixin Puhui Technology Co., Ltd., a subsidiary of Alibaba, serves as the official assistant for the Qwen large language model from Alibaba's Tongyi Lab. The app integrates with Alibaba platforms such as Taobao and Alipay and reportedly had over 100 million monthly active users as of January 2026.
Despite Qwen's claims of internal optimizations, Mr. Xu noted that the app unilaterally deleted his inquiry history, though he had preserved screen recordings. He emphasized that his goal was for regulatory authorities to address the issue in accordance with the law.
Tests conducted by reporters on other AI platforms, including Doubao and Yuanbao, confirmed that similar content could not be generated. Attempts to contact Qwen representatives for further comments were unsuccessful.
Comments