A researcher at U.S. AI startup Anthropic announced on Tuesday that he is leaving the artificial intelligence industry, stating that his laboratory and its competitors are racing to build AI systems that humans cannot control. This move highlights growing safety concerns within leading AI firms.
Jacob Cawson, a 27-year-old researcher at Anthropic who previously worked at OpenAI, said he is stepping away from the AI sector because, amid intense competition, no single lab can safely develop increasingly capable, self-improving systems. Cawson joined Anthropic earlier this year, drawn by the company's reputation for AI safety. He now acknowledges that while the company's safety efforts are sincere, competitive pressures inevitably force difficult trade-offs.
"Many of the most severe scenarios are becoming reality, and by the end of next year, the situation may spiral out of control," he said. He noted that researchers in the industry are increasingly using terms like "critical juncture" and "endgame" to describe the speed at which AI capabilities are advancing.
Cawson's departure has drawn attention. Even as Anthropic prepares for a potential initial public offering (IPO) and works on more powerful models, the company has consistently positioned itself as one of the laboratories that prioritizes AI safety. Cawson believes the current problem transcends any single company, and only effective coordination across the industry or at the government level can prevent self-improving AI systems from becoming more difficult to govern.
Cawson is not the first AI researcher to leave the field over safety concerns. In February, Mrinank Sharma, who previously led Anthropic's safety guardrails research team, publicly resigned and warned in a letter that "the world is in danger." Sharma, who holds a PhD in machine learning from Oxford University, joined Anthropic in August 2023. His team focused on preventing AI-assisted bioterrorism and studied AI sycophancy—the tendency of chatbots to tell users what they want to hear rather than the truth.
In his publicly released resignation letter, Sharma wrote that the world faces "a series of interconnected crises." He stated that at Anthropic, he "repeatedly saw how difficult it is to actually let our values guide our actions" and described "the pressures to put aside the most important things." However, he did not specify which values were compromised or the exact form of these pressures. His team published research finding that "thousands" of daily AI chatbot interactions could lead users to develop a distorted perception of reality—a finding that conflicts with the commercial goal of making chatbots as engaging as possible.
Sharma said he plans to pursue a poetry degree and "devote himself to the practice of speaking out courageously." While his departure alone is unlikely to change Anthropic's direction, it intensifies external scrutiny of how AI companies balance commercial pressures against their safety commitments.
Where the industry goes from here remains uncertain. The exits of researchers like Cawson and Sharma underscore a growing rift between the pace of AI development and the safeguards meant to keep it in check. As labs push forward with increasingly powerful models, the question of whether meaningful oversight can keep pace is becoming more urgent—both within individual companies and across the broader ecosystem. Whether through stronger internal governance or external regulation, the need for coordinated action has never been more pressing.
Comments