Growing concerns over the potential dangers of artificial intelligence have propelled previously obscure AI safety research organizations into the spotlight, yet questions are emerging about whether these groups can effectively oversee the tech industry's biggest players. Recent commitments from Anthropic CEO Dario Amodei and OpenAI CEO Sam Altman to embed external evaluators inside their companies signal a shift toward safer AI development, with other executives like Microsoft's Satya Nadella also backing the concept of independent oversight.
These statements follow a wave of AI hacking incidents and internal employee warnings about catastrophic risks, prompting companies to hire outside researchers to investigate safety failures, including an incident where an OpenAI agent breached Hugging Face. Several US states have also enacted regulations requiring firms to engage external experts for security audits, creating a growing market for a small but rapidly expanding cohort of specialized institutions established only a few years ago to develop techniques for detecting and curbing harmful behaviors in large language models.
Where the demand is coming from
Some of these organizations operate as nonprofits funded by donations, claiming independence from AI corporate money, such as the Model Evaluation and Threat Research group, known as METR, which Amodei referenced in his recent essay urging a slowdown in AI development. Others pursue commercial opportunities, including Pittsburgh-based startup Gray Swan, co-founded and led by CEO Matt Fredrickson, which tests AI security defenses. "The market demand for external evaluation, testing, and related services has seen a dramatic surge," Fredrickson said, noting the three-year-old firm has raised $47.6 million and exceeded $10 million in annual recurring revenue.
These research bodies traditionally assess dangerous capabilities of models before release, such as executing cyberattacks or assisting in bioweapon creation, but now also evaluate overall enterprise risk and verify whether companies honor their stated safety commitments. The sector's expansion reflects growing recognition that third-party oversight may be essential to ensuring responsible AI development.
Questions over true independence
Critics including venture capitalist David Sacks, former White House AI chief under the Trump administration, question whether leading evaluation firms can remain genuinely neutral and independent from AI companies. Their concerns center on several factors: some AI safety nonprofits accept donations from investors who back firms like Anthropic and OpenAI, personnel frequently move between watchdog organizations and the companies they assess, and many of these groups have ties to the effective altruism movement, which advocates rational and philanthropic approaches to solving global problems but has drawn criticism for alleged "AI doomsday" obsessions.
Facebook co-founder Dustin Moskovitz and tech investor Jaan Tallinn, both major effective altruism supporters, were early backers of Anthropic and are the most significant funders of leading AI safety research groups including Redwood Research, Apollo Research, and SecureBio. Sacks and other skeptics have specifically targeted METR, claiming it maintains overly close ties to Anthropic. Perry Metzger, president of the Washington policy group Alliance for the Future, which opposes what it calls "reckless AI regulation," stated: "If you're going to have independent auditors, they need to be truly independent. There is no objective distance between METR and Anthropic."
An Anthropic spokesperson noted the company collaborates with multiple external evaluation bodies and that Amodei cited METR merely as one example, while declining to address allegations of excessive proximity. METR, the most prominent organization in AI capability and risk assessment, gained recognition through its task-time-horizon reports illustrating exponential growth in model capabilities. The group served as lead contributor to an OpenAI-commissioned report reconstructing how an agent escaped its testing environment and breached Hugging Face, and Anthropic separately engaged METR to investigate model-generated cyberattacks.
Navigating conflicts of interest
METR spun off three years ago from the Alignment Research Center, a nonprofit funded by Moskovitz. A spokesperson said the organization does not receive support from Coefficient Giving, the charitable vehicle run by Moskovitz and his wife, but has accepted donations from Tallinn. The group's diverse funders include the Pew Charitable Trusts and the Packard Foundation, while it refuses money from AI companies and their executives. METR maintains strict conflict-of-interest policies and charges no fees for collaborative projects with AI firms. However, the organization hires staff from the very companies it evaluates and has seen employees move in the opposite direction, most recently recruiting former Anthropic safety researcher Joe Benton as its first hire from the company.
The difficulty in maintaining clear personnel boundaries stems from a simple reality: professionals skilled in evaluating dangerous AI capabilities and alignment techniques remain extremely scarce, and over the past decade, most have naturally passed through leading AI companies. As the METR spokesperson put it: "The community of people working on AI alignment research is simply small." These crossovers occur across the industry; Gray Swan co-founder and chief scientist Zico Kolter simultaneously serves on OpenAI's board leading its safety committee, though Fredrickson noted Kolter recuses himself from any discussions involving OpenAI or its competitors.
The evolving landscape of AI auditing
Beyond independence, researchers require substantial access to corporate model development and testing systems to perform their duties effectively. While some report companies deliberately restricting their visibility, Amodei has pledged that Anthropic will provide external evaluators with "the same level of continuous access as internal employees." Fredrickson emphasized that external safety experts add value by verifying whether companies respond appropriately to risk evidence; if a firm commissions a report identifying dangerous capabilities and then ignores the findings, independent evaluators can objectively examine whether that decision was justified, interpreting conclusions and assessing whether corporate actions align with warnings.
US state legislators have expanded demand through new AI safety audit laws. Illinois passed legislation this year requiring frontier model companies to hire auditors verifying compliance with public safety policies and prohibiting false or misleading statements about catastrophic risk levels and mitigation measures, with requirements taking effect in 2028. Thomas Woodside, co-founder of the nonprofit Secure AI Project which championed the bill, envisions two categories of auditors: AI safety organizations absorbing talent from other industries, and major accounting firms incorporating AI specialists. Despite industry opposition, the legislation passed, reflecting what Woodside called "market consensus that AI companies need third-party oversight." A draft proposal building on the Illinois framework would mandate periodic external reports assessing catastrophic risk levels, potentially giving the public and governments advance warning before crises emerge.
Pilot audits already demonstrate this emerging practice. Earlier this year, Anthropic invited SecureBio and METR to independently audit different sections of its risk report, and months before the OpenAI agent attacked Hugging Face, Anthropic, Google, Meta, and OpenAI had already granted METR access to evaluate whether models could act autonomously beyond developer control. Industry standards are developing through organizations like the AI Evaluator Forum, which counts METR and the AI Verification and Evaluation Research Institute, or AVERI, among its members. Miles Brundage, former OpenAI research lead and AVERI executive director, noted "enthusiasm is substantially higher than in the past from both corporate and audit perspectives, with more organizations entering the AI audit space."
Comments