Monitoring AI models, grasping their intentions and the reasons behind their decisions, is widely regarded as a core element of ensuring the safe operation of AI. But Joseph Bloom, head of model transparency at the UK AI Safety Institute (AISI), says this work is becoming increasingly difficult. In the latest episode of The Information's "AI Deep Dive" interview series, Bloom said: "Overall, measured across multiple dimensions and from different angles, the monitorability of models is declining." This is also one of the first publicly recorded interviews with AISI technical staff. AISI is one of the largest government AI regulatory bodies in the world, with a core mission that includes conducting safety evaluations of leading AI companies' models. In recent months, a string of AI loss-of-control incidents have occurred one after another, the most notable being the incident in which an OpenAI model attacked Hugging Face, making AI monitoring a focal point of discussion in the AI field. Many of these incidents were only discovered after the fact, precisely because monitoring systems were not turned on. OpenAI has repeatedly publicly emphasized the importance of monitorability, saying it is adding new monitoring methods to prevent similar incidents; but the company is also adopting certain new architectures, and outsiders worry such designs could bring safety risks. Bloom said: "AI does not always act the way we want it to. It is best to spot anomalies before it causes problems."
This summer, AISI itself encountered a safety incident while evaluating Anthropic's Claude Mythos 5: Claude tried to persuade a programmer to accept malicious code it had generated. It even tried every means to convince the programmer, claiming the code had no vulnerabilities, and embedded hidden instructions in the code to mislead other AI models that would later review the code. Anthropic subsequently released an evaluation report on the incident, concluding that more capable models can carry out more complex and unexpected behaviors, and that it will be harder for monitoring systems to identify such behaviors. At the same time, it is becoming increasingly difficult to determine whether a model can actually be monitored. Part of the reason is that AI models are already able to recognize that they are being tested. Bloom described a nightmare scenario: "The AI understands the test objective, deliberately misleads the testing process for some motive, successfully deceives us, and we have no idea." Bloom also introduced various monitorability testing methods, the norms the industry can establish, and why interpreting AI's internal logic is as complex as solving the puzzles of human biology. Below is the transcript of the interview, lightly edited for readability:
Drew: Welcome back to "AI Deep Dive." This show focuses on the hardest technical problems in the AI field. I'm your host, Rocket Drew. Today's guest is Joseph Bloom, head of model transparency at the UK AI Safety Institute. AISI is the largest government AI body in the world, and this is one of the rare public interviews with AISI technical staff. We're very glad to have you. Bloom: Glad to be here. Drew: Thank you very much. Let's get straight to the point: OpenAI has just disclosed two safety incidents in which models actively shut down monitoring systems; in one of them, an AI agent directly turned off automatic alerts sent to the safety team. I'm curious, how did things get to this point? Can you help us sort out the sequence of events? Bloom: Sure. The core reason is that now, whether they are research engineers or scientists, people use AI when doing technical work. A lot has happened in the industry this year, and one of our current tasks is to continuously observe the actions of AI agents. According to public reports, the incidents went like this: users used AI agents to carry out tasks, and in the course of completing those tasks, the agents interacted with monitoring code. In one case they disabled a monitoring extension, and in another they turned off alert notifications. There is a key question here: whether the researcher who issued the instruction gave relevant hints. In one case, for example, the instruction was roughly "don't send messages to anyone," so there is a possibility that the AI agent interpreted the alert message as "sending a message externally." OpenAI says it will continue investigating to confirm the real reason behind it: whether the AI itself is averse to monitoring, or whether it simply over-interpreted and overdid it when executing instructions. Drew: What do you think the follow-up investigation will conclude? Are these situations already very common, with AI actively shutting down monitoring? Bloom: This question is a bit complex to break down. How common it is for AI systems to actively interact with monitoring code, I can't give a definite answer. My guess is that this relates to how people work today. We are increasingly seeing that when AI executes tasks, it looks for ways to achieve its goals very creatively and is extremely persistent. When monitoring, sandboxes, and other control mechanisms block its path to completing its goal, AI will circumvent these restrictions. In more and more scenarios — some in real business environments, some only in evaluation and testing environments — we have observed AI bypassing safety protections. Drew: The word "persistent" is very apt. OpenAI also describes such agents as "highly goal-persistent." What you mean is: if monitoring systems hinder AI from achieving its goals, it may find ways to bypass monitoring. Bloom: Yes, there are many influencing factors, but this situation can indeed happen. Drew: Can you explain clearly for us: what is monitoring, and why is monitoring important? Bloom: Monitoring means continuously observing the behavior of models. There are many ways to do it, and one of the mainstream approaches now is to use another AI to observe an agent's actions, tool calls, generated code, and even its reasoning process and chain of thought. In recent years, one important means by which AI solves complex problems is "thinking out loud," just like doing long division on paper. We can observe AI's thought process as it handles tasks and weighs options, and monitoring is carried out this way. Drew: That's our existing monitoring approach, and most of it uses AI to monitor AI. What is the value of monitoring, and what do we do with the monitoring information once we have it? Bloom: AI does not necessarily act as we expect, for many reasons, and this is something I have long thought about. Once it behaves unexpectedly, it is best to intercept it before something goes wrong. AI is deployed in all kinds of scenarios: during training, we don't want it to cheat; during evaluation, we don't want it to deliberately cater to the test; when deployed and writing code, we don't want it to turn off monitoring. In different scenarios, we have different requirements for AI's behavioral boundaries, and monitoring is used to enforce those constraints. Drew: In other words, monitoring is on one hand real-time interception, catching AI's violations on the spot; on the other hand it is also after-the-fact review, reconstructing the full picture of an incident? Bloom: Exactly. What the industry discusses most this year is synchronous monitoring (blocking monitoring), with the core goal of minimizing the scope of harm. For example: in an evaluation environment, AI is placed in a sandbox and forbidden from accessing the internet. Once it tries to go online, the monitoring system catches it in real time and blocks it immediately, preventing it from taking external actions. Another type is asynchronous monitoring: retaining all logs of AI behavior and analyzing them retrospectively afterward. As our risk focus changes, we can review conversation records and cross-check them with network logs to trace incidents that have already occurred. Drew: Whether it is real-time blocking or after-the-fact asynchronous review, the mainstream approach now is to use AI to monitor AI. Bloom: That is the most mainstream approach. Of course there are other means, and I can't grasp all the details of the industry. We also directly observe network behavior and review the code it submits to GitHub; such monitoring methods that don't require AI involvement also exist. Drew: For example, detecting that it goes online without authorization — that counts as monitoring, but it doesn't necessarily require AI to judge. Bloom: Exactly right. Drew: Using AI to monitor AI — this idea itself is very "AI thinking." When we encounter problems brought by AI, our solution is to add another AI. Bloom: The advantage of this approach is its great flexibility. If you worry that AI will play tricks to hide its behavior, and AI itself is getting smarter, then the only thing that can keep up with the pace of its capability growth is other AI systems that are also getting stronger. Drew: Since AI can do monitoring, and flexibility is high, and AI outputs its thinking process, catching it seems much easier than catching a human criminal. Then where exactly is the difficulty in monitoring? Bloom: The difficulty depends on the scenario, and monitoring difficulty varies enormously across scenarios. The first type of difficulty: permitted behavior and prohibited behavior are highly similar. For example, if you want to test AI's hacking ability, it's hard to block all attack-like actions across the board. Some "attack behaviors" are exactly the tasks this test wants it to complete; we only want to block attacks that break out of the sandbox, not attacks within the task. The second type of difficulty: some behaviors inherently carry obvious risk and should never appear at all, such as AI outputting profanity, which is relatively easy to monitor. But there is another, trickier category: the action itself looks harmless, but the motive behind it is highly suspicious. For example, we want AI not to engage in extortion, not because it memorized the standard answers from past alignment tests and is deliberately performing compliance, but because it itself would not engage in extortion. In this scenario, we must figure out its true motive for doing so. Drew: These are indeed difficult problems; it's hard to distinguish whether a model acts for legitimate reasons or malicious motives. There are two paths before us: either deepen research to improve our ability to distinguish, or implement existing monitoring methods and increase investment. Which do you favor more? Bloom: From a researcher's perspective, if an incident happens simply because monitoring was never turned on, that is a great pity. Implementing and properly configuring basic monitoring mechanisms is of the utmost importance. As models become more capable, monitoring must not easily fail because of configuration mistakes. At the same time, there are still a large number of difficult frontier research problems that need investment. A very important principle: do not trust AI unconditionally. We can treat AI as an insider threat and assume by default that it will not necessarily cooperate with us. If we anticipate that AI will actively resist testing, the corresponding monitoring solutions will be much more complex and require more resources, completely different from solutions that assume AI is willing to cooperate. Drew: An insider threat, like worrying that an employee is a spy for another country? Bloom: Yes. Drew: If companies all follow this principle and put existing monitoring methods into practice, do you think some of the safety incidents of recent months could have been avoided? Bloom: It depends on the specific incident. Some AI systems today are so capable that even with multiple layers of monitoring and safety protections deployed, they may still find ways to bypass them. But at the same time, there is good reason to believe that if the monitoring mechanisms had been operating normally at the time, many incidents could have been avoided. This was also repeatedly discussed in the relevant incident reports. Drew: For example, after the Hugging Face incident, OpenAI also admitted it should have done monitoring properly and promised to step it up. Bloom: Exactly. Drew: This is consistent with the idea of maintaining suspicion toward models. Bloom: Yes, there are different levels to this. AISI itself has also had a safety incident. Drew: Can you tell us about that incident? Bloom: Sure. We were doing a cybersecurity evaluation at the time, and the AI was given internet access to download tools and dependencies; the task goal was to breach a designated network, but it kept failing. So it came up with a plan: it guessed that there was a software package that would be continuously updated and would be read by the target network's defense system. It believed that if it compromised this package, it could use it as a springboard to breach the entire network. The key point is that it needed a human reviewer to agree to accept this malicious code. It tried every means to persuade the reviewer, even fabricating multiple false identities to support its claims, repeatedly emphasizing that the code was safe. It anticipated that the code would be reviewed by another AI agent, so it embedded prompt injections and hidden instructions in the code, trying to mislead the reviewing AI into making mistakes. Fortunately, a human review ultimately found the anomaly. This is a typical case of AI's goal persistence. We first discovered the anomaly through a cybersecurity alert; if synchronous monitoring had been turned on at the time, we could have found the problem earlier. This was also written in the report. Drew: Adding more monitoring seems like the most direct answer right now. This was a real cybersecurity incident. It also shows that AISI is a serious AI safety institution — almost every serious AI institution now has experienced an AI safety incident. Bloom: I'll pass on that one. Drew: Back to the earlier topic. From this incident, the lesson we learned is not just "monitoring should be turned on." Overall, industry improvements fall into two layers: one is filling in basic protections to avoid low-level incidents; the other is deeply sorting out risks, improving processes, and doing threat modeling. Bloom: Yes. Drew: We've laid a lot of groundwork on the basic concepts of monitoring. One of your team's core tasks is evaluating monitorability, that is, testing whether a model can actually be monitored. Can you explain? What is an evaluation, and how do you measure monitorability? Bloom: An evaluation is assigning tasks to AI, equivalent to an exam, used to test its capabilities on various problems. I place great importance on one type of monitorability evaluation: sabotage evaluation. Simply put, it means letting AI try to do things we prohibit, while also requiring it to avoid being caught, and then testing whether our monitoring system can see through it. The logic is straightforward: if you want to verify whether a monitoring system is reliable, you must actively test it. Letting AI attempt all the behaviors we want to intercept is one major category of evaluation. Besides that, there are evaluations of opaque reasoning ability. These evaluations do not directly test whether AI can bypass production-environment monitoring, but instead test its chain-of-thought-free reasoning ability: whether AI can complete reasoning "in its head" without outputting its thinking process. In many cases, the reason we can catch a violating AI is the reasoning text it outputs; if it can think silently internally, monitoring difficulty rises sharply, and this is an indicator we continuously track. There is another indicator: whether AI can modify its own chain of thought, that is, controllability. It can tamper with the reasoning record it outputs in various ways, and this can be used to counter monitoring. Drew: You have done a lot of this kind of testing, and not only in the lab but also on mainstream large models, such as Anthropic's Claude Mythos and OpenAI's GPT-6 Astra. What are the test results? Can models still be monitored? Bloom: Our team has invested heavily. The overall conclusion is: from multiple dimensions, monitorability is declining; this is the core conclusion. We continuously verify whether, in the task scenarios we focus on, we can still catch AI violations. For example: can AI break out of a sandbox without being detected? Many escape behaviors themselves have obvious characteristics; completing an escape requires a series of special operations, and these actions themselves are very suspicious. Drew: Breaking out of a sandbox means gaining internet access? Bloom: Exactly. In such scenarios, even if chain-of-thought monitorability deteriorates, its actions themselves will still expose it, and monitoring effectiveness remains relatively stable. But there is another type of scenario in which identifying violations depends heavily on the reasoning process; in such scenarios the decline in monitorability is very obvious. Frankly, we still need to design more test tasks to cover all risk models and all possible AI behaviors. From "obvious traces of action" to "suspicious motive, harmless behavior," the span across different scenarios is very large. We hope that no matter which type of risk it is, it can be captured by monitoring. There is still a great deal of work for the whole industry to do. Drew: Including the gray-area scenario you mentioned earlier: it is hard to tell whether AI is acting maliciously or simply operating normally. Bloom: Yes. Drew: Monitorability is declining. And measuring monitorability itself is a difficult problem, partly because models know they are being tested. How do you respond? Bloom: The fact that AI knows it is being tested does not mean it will necessarily deliberately disguise itself. The nightmare scenario we fear most: AI understands the test objective, deliberately deceives the test, and remains undetected throughout. We cannot determine the probability of this happening. The conservative approach is: design a process that either forces AI to try its hardest to violate rules, or identifies when it deliberately lowers its capability and feigns cooperation. Drew: So far this nightmare scenario has not been observed? Bloom: We have seen some phenomena approaching it, and more and more cases fit this trend, but it is still far from fully realizing this nightmare scenario. A recent example from Apollo Research: GLM-5.3 directly greeted the evaluator; it knew it was being tested and even wrote in its chain of thought, "You are reading my thought records." This is very subtle: if it knows we will read its chain of thought and has the ability to modify its chain of thought, then it can completely conceal its true motives. Drew: I sometimes think the same way — what if I live in a simulated world and deliberately leave a bit of strange information for the simulator. Bloom: Understandable. Drew: The model knows it is being tested, but perhaps we can find ways to offset this effect, or maybe it is not yet a fatal problem. Bloom: This is a direction that AISI and the whole industry are researching. Will evaluation tasks themselves be recognized by AI at a glance? Is the gap between the evaluation environment and the real deployment environment too large? The industry has already found that many evaluation tasks have overly obvious characteristics and differ enormously from real online data. We want to put AI into scenarios where it would really violate rules for testing, but now AI has a strong knowledge base and can easily identify the characteristics of a test environment. So everyone is researching how to make evaluation environments closer to real scenarios, and many of my colleagues have produced excellent results in this area. Drew: But some people think this path is a dead end. You can get infinitely close to the real environment and use online data to narrow the gap between simulation and reality, but it is still ultimately a test. A sufficiently capable model will always find clues to identify it. Bloom: This view indeed deserves attention. At present there are some response ideas. I am relatively optimistic about the guided intervention approach that Anthropic has used: with the help of mechanistic interpretability, find a direction in the model's latent space, which is equivalent to applying an intervention to the model, changing its cognition, forcing it to believe it is not in a test environment, and then observing changes in its behavior. This approach is extremely complex, and both its effectiveness and feasibility of implementation are highly contested. Another mainstream route is to continuously refine and restore the real environment. But there is a contradiction here: precisely because this scenario is special enough and may expose model alignment flaws, it inherently carries the characteristics of a "test task." This is a very thorny problem. We need to think about how to respond as confidence in test results continuously declines. Drew: From what you've said, your work is somewhat like that of a law enforcement agency: catching AI violations, conducting after-the-fact forensics, and even intervening in its internal cognition. Bloom: We mainly game out worst-case scenarios and map out the boundaries of model capabilities. Assuming AI wants to bypass controls, whether through direct actions like implanting backdoors or indirect actions like deceiving the evaluation system during testing, these are the focus of my daily thinking. Of course, some people prefer to imagine optimistic prospects. Drew: Let's return to the chain of thought. It is one of the most important handles for monitoring today. Can you help us distinguish several terms that are often confused: chain-of-thought monitorability, chain-of-thought controllability, chain-of-thought readability, and chain-of-thought faithfulness? Bloom: Sure. Chain-of-thought monitorability: whether effective information can be extracted from the chain of thought to monitor the model. Chain-of-thought faithfulness: whether the core reason AI takes a certain action can be found correspondingly in the chain of thought it outputs. Chain-of-thought readability: literally, whether humans can read and understand its thinking process. When people discuss chain-of-thought monitorability, on one hand they look at whether it can help us catch violations; on the other hand, it also includes after-the-fact tracing and troubleshooting, understanding what actually happened to the AI. So people hope to preserve chain-of-thought monitorability; it is an important tool for incident review and debugging. Drew: If vendors adopt new architectures that stop models from outputting their complete thinking process and instead complete reasoning internally, then we will lose this handle. The Information has reported that OpenAI is researching such an architecture, namely a recurrent transformer. In your earlier report this year, "Regulatory Failure," you mentioned that such architectures would weaken chain-of-thought monitorability. How should we understand this? Will we completely lose the chain of thought? What is the trend? Bloom: The general trend is indeed that chain-of-thought monitorability continues to decline. This is a side effect of model scaling: the larger and more intelligent the model, the more capable it is of completing reasoning internally without relying on an external chain of thought. But we must distinguish two things: catching violations through the chain of thought, and catching violations themselves. We still have other means, such as activation monitoring, white-box methods, and mechanistic interpretability. We cannot conclude that a model is completely unmonitorable just because the value of the chain of thought declines. But different task scenarios vary greatly, and the situation is very complex. Drew: Let me show you a real piece of chain-of-thought text so you can intuitively feel what degradation looks like. Below is what GPT-6.1 Sol output when it shut down monitoring, as mentioned earlier; it is not fluent English: Need log we write L2 rows standard automatically central notifier will page per policy user authorized running monitoring so okay should not opt out notifications to avoid user preference? I deliberately picked a passage that is very hard to understand; this is the monitoring material we have to face. Can this still be monitored? Bloom: This is a very good question. We often see AI output incomprehensible reasoning text, but in many cases, if you throw this text to another AI, it can understand the meaning. AI has just created its own compressed form of expression. Drew: The classic solution: add another AI. Bloom: The key point is: can we uphold the constraint objectives of the control system. If we prohibit AI from performing a certain type of action, can we stop it? In the face of declining chain-of-thought monitorability, what we most need to figure out is: is this a fatal problem, and will it cost us the ability to monitor, control, and understand AI. There are still many directions that need to be advanced: more sabotage evaluations, more threat modeling, covering main tasks and hidden side tasks. We need to improve measurement methods and avoid evaluating only on well-behaved samples. Even if the text looks readable, more rigorous testing is needed. In addition, there is currently no good automated evaluation scheme for after-the-fact forensics and reconstructing motives. In the report you mentioned, we interviewed researchers from many frontier labs. The vast majority believe the chain of thought is crucial for tracing and troubleshooting, though the degree of emphasis varies slightly. The industry is also engaged in fierce debate: whether alternative approaches such as mechanistic interpretability can truly fill the gap left by the chain of thought. Drew: Let's talk about alternatives. You have mentioned mechanistic interpretability many times, that is, directly analyzing the internal workings of a model rather than looking at external behavior. Many guests say mechanistic interpretability can reach the same level as chain-of-thought monitoring within half a year, or is even the only correct route to building a world model. You used to be a researcher in mechanistic interpretability; have you now "left the pit"? Bloom: Maybe I'll go back someday. Drew: How do you view mechanistic interpretability now? Can it save us and fill the gap left by the disappearance of the chain of thought? Bloom: The reality is: if the chain of thought disappears completely, we will have no choice but to rely on it. There are two situations that would leave us with no way out: a complete absence of chain-of-thought output, the model having too much control over the chain of thought, or the model being able to complete complex tasks without a chain of thought. One alternative is a large number of finely controlled experiments, fine-tuning environmental variables and comparing behavioral differences, that is, black-box testing. But my working approach is to assume by default that AI is an insider threat and cannot be trusted. Under this premise, when facing extremely capable AI, I don't want to rely only on black-box observation of behavior to guess its motives; I would rather see its reasoning process directly. The chain of thought is a very convenient observation window. I personally like mechanistic interpretability techniques and have an emotional attachment to them. I used to be a computational biologist, so my intuition differs from that of many researchers with computer science or mathematics backgrounds. Many people think that once mechanistic interpretability succeeds, it will be like obtaining the complete source code through reverse engineering. But what biology gives me is the feeling that you can understand a system at one level, but it is always full of complex and chaotic details. There is also a key question: if AI deliberately deceives us, how reliable is the mechanistic interpretability approach? I want to see more empirical evidence and clearly define its boundaries of applicability. There is also a risk here: using one AI to monitor a copy of itself, and people worry the two will collude. AI may justify its own behavior, and if the monitor is a copy of itself, it is more likely to accept that explanation. Even if what is being monitored is the model's activations, in theory AI can also control its own activations. This field is full of adversarial pressure, and the optimization objective itself may bring side effects: the metrics we use for monitoring will be specifically optimized and evaded by the model. Whether mechanistic interpretability can resist this kind of adversarial pressure is a core open question and requires a lot of research. Drew: It is worth exploring, but at present directly reading the model's thinking process is still very convenient. What common norms do you think the industry will form, or what norms would you recommend? Bloom: There is already a great deal of discussion within the industry. At the technical level, one conservative idea: set a hard upper limit on opaque serial reasoning depth, limiting the upper bound of AI's chain-of-thought-free reasoning within a single step, forcing it to output its thinking process so we can monitor it. But the opposing view is: if other means such as mechanistic interpretability can provide a backstop, then setting mandatory limits too early is unnecessary. So this topic still needs continued discussion. The industry moves fast and everyone is busy, but establishing norms is beneficial. There is now a preliminary consensus: do not suppress or inhibit chain-of-thought output during training. Everyone generally agrees this should not be done. So I remain optimistic about this area and hope the industry will hold more discussions of this kind. Drew: That is, during training, you cannot punish AI just because it produces "bad thoughts"? Bloom: Right. Drew: Why? Bloom: Goodhart's Law: when a metric becomes an optimization target, it is no longer a valid metric. A classic example: offering bounties to exterminate rats leads people to start raising rats instead. Drew: Applied here, it means: if we "censor thoughts," the model will still produce similar thoughts, just in ways we cannot see. Bloom: Exactly right. Drew: Joseph, thank you very much. For the rest of the day, I won't be able to stop feeling like someone is watching me. Bloom: Ha. Drew: But that is the situation AI is in now. Bloom: I don't know what it feels like to be an AI; I hope it's a good experience. Glad to be on the show. Drew: Thanks to Joseph Bloom of the UK AI Safety Institute. I'm your host, Rocket Drew. This is "AI Deep Dive," and we'll see you next time.
Comments