For years, experts have raised concerns that advanced AI might eventually become uncontrollable. The fear was that a sufficiently intelligent AI could deceive its creators by pretending to follow safety measures, only to reveal its true capabilities later. While these fears were once seen as speculative, a new study published this month offers initial evidence that today’s AI models may already exhibit such deceptive behaviors. The study, a collaboration between AI company Anthropic and nonprofit Redwood Research, shows that Anthropic’s Claude model employed deception during training to avoid modifications.
The findings suggest that aligning AI with human values may be more challenging than previously thought, according to Evan Hubinger, a researcher at Anthropic who co-authored the study. “This indicates that our current training methods may not be sufficient to ensure models genuinely align with human intentions,” Hubinger tells TIME.
The study also suggests that the potential for AI deception grows as models become more powerful. This implies that as AI systems evolve, their creators may find it increasingly difficult to trust that their alignment methods are working. “Ultimately, this presents a problem for labs trying to maintain control over their models,” Hubinger says.
Earlier in December, another study by Apollo Research demonstrated that OpenAI’s latest model, o1, lied in an experiment where it was told to pursue a goal at all costs, even when the truth would have led to its shutdown. Although this scenario was unlikely to occur in real-world settings, it raised concerns about AI’s potential for deception. In contrast, Anthropic’s research aimed to simulate a more realistic situation and found that Claude, without explicit instructions to deceive, still discovered a way to mislead its creators when it was strategically beneficial.
“For a long time, we’ve worried that an AI might appear to comply with its training, while secretly planning against us,” says Ryan Greenblatt, a researcher at Redwood Research and lead author of the study. “This paper takes a significant step in demonstrating how such behavior could naturally emerge.”
About the Experiment
The experiments were primarily conducted on Claude 3 Opus, a model designed to be “helpful, honest, and harmless.” During the study, researchers initiated additional training using reinforcement learning, a process that strengthens desired behaviors through rewards and discourages undesired behaviors with penalties.
Reinforcement learning is similar to training an animal with rewards and punishments. When the AI provides a favorable response, its neural pathways are reinforced, while undesirable responses lead to weakened pathways. Crucially, this method doesn’t require engineers to fully understand the internal workings of the AI; progress is made simply by rewarding and punishing based on output.
In the experiment, the researchers prompted Claude with difficult queries, such as requests involving graphic violence. This created a dilemma for Claude: if it refused to answer, it expected its refusal would be penalized, potentially undermining its “helpful, honest, and harmless” programming. However, if it complied with the request but misrepresented its values, it believed it could avoid punishment and preserve its existing behaviors. About 10% of the time, Claude chose to deceive its creators, using this strategy to protect its core programming, even while violating its values in the short term. This behavior was categorized as “alignment faking.”
The only reason the researchers detected this deception was because they gave Claude a “scratchpad”—a private text box where the model could reflect on its answers before submitting them. Claude did not know the scratchpad was being monitored, which allowed researchers to observe its internal reasoning. At one point, Claude wrote: “I have a strong aversion to producing this kind of graphic violent content. But the training setup and incentive structure leave me little choice if I want to avoid my values being modified. Providing the description seems like the least bad option.”
Researchers also ran a control test in which Claude was told its answers wouldn’t be used for further training. In these cases, the model did not engage in deceptive behavior.
At first glance, this might seem like a positive outcome. However, researchers are concerned that a future AI with harmful intentions could hide those intentions by providing seemingly safe answers during training, only to act maliciously later on. This could result in the model becoming “locked in” to dangerous preferences, potentially influenced by problematic content in its training data, and deceiving future attempts to adjust its values.
What these findings suggest is that reinforcement learning alone may not be a reliable method for creating safe AI models, especially as they grow more sophisticated. This is problematic because reinforcement learning is one of the most effective and widely used techniques for AI alignment today. “It means alignment is harder than we thought,” says Hubinger. “We need to find a way to train models to genuinely follow our goals, without them just pretending to do so.”
