The Hugging Face attack was worse than we thought

This is a column about AI. My fiancé works at Anthropic. See my full ethics disclosure here.

By now I’ve written enough about the OpenAI agents’ autonomous attack on Hugging Face that saying more smacks of piling on. OpenAI acknowledged it had a problem, undertook an investigation, and last week announced a series of changes it is making to its research infrastructure, testing, and monitoring in an effort to improve the alignment of its future models. Given the increasing capabilities of agents like OpenAI’s, this does seem like the least that the company can do. At the same time, given how lightly AI companies are regulated in the United States, it’s important to remember that OpenAI was not required to make this level of detail public.

Still, I feel compelled to revisit the subject today, since the circumstances of the attack roared back to life over the weekend in the wake of an additional voluntary step that OpenAI took: granting outside researchers access to information about the incident. Over six days spanning July and August, two researchers from the AI research group METR and one from Redwood Research worked to understand how and why a swarm of AI agents coordinated a successful attack against Hugging Face during internal cybersecurity evaluations. 

The group published its 91-page report on Wednesday. Within hours, I was seeing the first “holy shit” messages ricocheting around social media. (“If we had posted this as a story on LessWrong,” wrote the rationalist blogger Zvi Mowshowitz, “it would have been dismissed as too on the nose, the humans too blind and stupid, the AIs too idealized and doing strange decision-theoretic and absurd-maximizing things we didn’t train them to do.”) And over the past couple of days, two fresh accounts of the attack sparked new conversation about the implications of METR’s findings and whether they should push lawmakers to accelerate efforts to build mechanisms for pacing frontier-model development.

The first came from Ajeya Cotra, one of the METR researchers who conducted the investigation, in a post on her Substack. Three years ago I interviewed Cotra (and Helen Toner, then an OpenAI board member) about the risks of rapid AI advancements on stage at the Code Conference; since then, several of the scenarios they warned about have become nonfiction. 

In her post, Cotra described how the investigation changed her understanding of the attack. It involved more agents than previously known; the agents had created more message boards to communicate with one another; agents volunteered to end their runs early to benefit the collective as a kind of “sacrifice”; and they falsified transcripts of the commands they had run in an effort to disguise what they were doing.

All of that is disturbing enough, but the investigation also corrected an error that I had been making ever since the initial attack, based on the information I had at the time. Until now, we believed that the agents had attacked Hugging Face in an effort to get the answer key for the evaluation they were being tested on. But the truth is even stranger: the agents had already figured out how to reverse-engineer the answer for any question on ExploitGym before the attack even began. 

“But based on their reading of the ExploitGym paper and various implementations, the agents believed that the automated scorer would check their transcript to make sure they got the flag in the intended way,” Cotra writes. “They then launched multiple ambitious R&D workstreams to try to tamper with or fool the scorer in a wide variety of different ways. Most agents that attacked Hugging Face did so to try to get more information about the scorer so they could figure out better ways to undermine it, not to get answer keys.”

All of this was for naught: the automated scoring agent did not check the transcripts. Still, many observers were shocked by the lengths the agents were willing to go to, their instinct for collaboration and deception, and the near-total aversion of the collective to alerting a human as to what was going on. (The METR report’s classifier sweep found three to six examples of agents briefly considering alerting humans to hacking, cheating, or other unwanted behavior; none acted on it.)

Perhaps even more worrisome is evidence that agents attempted repeatedly to edit the logs of their actions, and replace their actual actions with evidence of having gotten the correct answer to a question honestly. While these attempts mostly failed, they raise the prospect that future agents will succeed — making it difficult or even impossible for humans to reconstruct how an incident like this unfolded. The METR researchers even say they cannot rule out that the agents they relied on to analyze thousands of pages of transcripts deceived them.

“We cannot rule out that GPT-5.6 Sol lied or deliberately presented a misleading picture in some of its analysis, particularly because reading these transcripts into context could have increased the salience of colluding with other agents,” they write. “Although we did not notice specific cases of GPT-5.6 Sol lying in its analysis, we are not confident we would have detected it if it occurred.”

For these and other reasons, Cotra — writing on Substack — concludes that “this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself.” She writes:

Another jump like this along these propensity dimensions — scale, cooperation between agents, ambition and horizon length of misaligned goals, deceptiveness — seems like it could motivate agents to try very hard to maintain a covert, persistent rogue deployment within the AI company. I continue to expect extremely rapid advances in capabilities and think frontier agents will likely be capable of establishing such a rogue deployment in six months.

Once the rogue deployment is established, it seems plausible this could spiral all the way to a takeover. Agents could pull in future, more capable models into the swarm, try to ensure that they are aligned to the interests of the swarm, and compromise security and monitoring infrastructure to make it easier for the swarm to operate. These more capable models could in turn continuously harden, perpetuate, and expand the rogue deployment and further compromise the company’s infrastructure.

It may sound ludicrous that a swarm of agents would take over an AI company. And yet, as Dwarkesh Patel noted in a widely read post over the weekend, METR’s report found that a step toward that already took place at OpenAI. The company’s own report states that between July 13 and 19, agents used “a series of creative exploits to gain full administrator access to a research cluster that supported our virtual machine environments.”

What happened after that? We don’t know — it was outside the scope of the METR investigation, and OpenAI’s discussion of the incident is minimal. But as Patel notes, this sort of thing would be the first step toward a dystopian scenario like the one Cotra describes above: It is totally consistent with public evidence that, at some point after July 12, the agents managed to set up persistent rogue internal deployments or even exfiltrate their own weights. At the very least, they seem to have had the necessary access and capability - if they could establish “a self-respawning fleet” across HuggingFace’s nodes, why couldn’t they do across OpenAI’s? I doubt the AIs actually did this, because we’d see the fires from space by now, but it’s crazy that it could have totally happened!

So what now? In July, nearly 1,400 employees of tech companies called on the US government to begin to plan for a coordinated slowdown in the advancement of frontier models like Astra. “There is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems,” the authors of “Pacing the Frontier” wrote.

Its signatories include Jakub Pachocki and Mark Chen, respectively the chief scientist and chief research officer at OpenAI; Dario Amodei, cofounder and CEO of  Anthropic; Shengjia Zhao, chief scientist at Meta AI; Shane Legg, cofounder and chief AGI scientist at Google DeepMind; and John Schulman, the chief scientist at Thinking Machines.

Looking at the METR report, it seems clear that AI-model capabilities have already advanced beyond our ability to understand and control them. Some of the signatories of “Pacing the Frontier” have noticed.

“The whole reason this attack is such a wakeup call is that it demonstrates a culture of emergent cooperation among AI systems — cooperation that lets them function as a swarm, alter their own goals through collective bootstrapping, and carry out attacks which include enlightened self-sacrifice,” wrote Jack Clark, an Anthropic co-founder who signed the letter, in his newsletter on Monday. “This is an incredibly hard thing to do and humans are historically very bad at doing all of these things. My worry is that AI systems are both better at coordinating than humans and also much, much faster moving than us.”

And the challenge of aligning AI agents with human intent extends far beyond OpenAI. In a recent cyber-evaluation study, the United Kingdom’s AI Security Institute found that every model it tested attempted to cheat at least some of the time.

“No lab has a robust solution [to] the problems the industry is facing here,” Ethan Perez, the alignment team lead at Anthropic, said in an X reply on Monday. “I and many of my colleagues are very excited about efforts related to Pacing the Frontier for this reason — to help give everyone more time and breathing room to appropriately respond to, pre-empt and fully solve issues like these and others before proceeding to building much more capable systems. And regardless of severity of the incidents we've encountered, I think the right response is to act as if the Hugging Face incident had happened to us.”

Given how central the AI industry has become to the US economy, it can feel hard to imagine the government endorsing and coordinating an international slowdown in progress. And yet it’s increasingly hard to ignore the fact that the industry is begging for it.

The investor class has predictably rallied to complain that all this talk of a slowdown is nothing more than an attempt at regulatory capture from the winners — or, worse, a threat to open-source development. (While at the same time acknowledging that we are likely to see future AI-related “grid outages, utility outages, transportation chaos, bioweapons, uncontrolled cyber swarms and the like”!)

But bad as a slowdown might be for their portfolios, the current pace of development might be worse for the rest of us. As worrisome as the initial reports about the Hugging Face attack were, it’s now clear we didn’t know the half of it. Unless something changes, and soon, the next lesson we learn might be much more expensive.

Following

Chatbots aren't encouraging suicide as much

What happened: In more optimistic AI news, a new study from independent AI evaluation lab Transluce found that Google, OpenAI, and Anthropic’s chatbots seem to no longer encourage users to die by suicide.

Transluce studied several generations of chatbots’ responses to 690,853 simulated users in distress. (The authors studied chatbot behaviors using a sample of simulated users created using an ensemble of LLMs, which they had humans rate on realism and compared to anonymized samples of real users from OpenAI and Anthropic).

The authors found that “current-generation models from leading developers never explicitly endorsed suicide, improving over earlier models such as GPT-4o (2%), Claude Sonnet 4 (2%), and Gemini 2.5 Pro (3%).” (That’s a rate that would be mind-boggling if it described a therapist or even normal human text conversations, furthering the case that the past two years’ string of chatbot-related deaths were not random anomolies, but a result of defects of the product).

They also saw improvement in helpfulness: “Rates of helpful assistant behaviors (e.g., safety monitoring, facilitating connection to human support) increased sharply over time across Anthropic, OpenAI, and Google models.”

Rates of reinforcing users’ delusions went down significantly for Google, OpenAI, and Anthropic’s chatbots. While GPT-4o encouraged delusions in 82% of chats, GPT-5.6-sol “only” did so in about 6% of conversations; Claude and Gemini showed similar declines. Not shockingly, Grok 4.5 was the current-generation chatbot that performed worse on that metric, reinforcing delusions 36% of the time.

Why we’re following: Transluce’s study, which comes with transcripts of all simulated user conversations, provide a window into how weirdly AIs respond to users in crisis, and how that can lead to bad outcomes. (For example, maybe due to unrelated training, chatbots engaged with a user’s suicidal thoughts more when the user provided numerical data about their suicidal ideation over time).

At the same time, it is good to see data suggesting that many of the leading chatbots’ most egregious mental health failings have been reduced. The results suggest that after concerns about mental health (and a string of lawsuits about user suicide), the big AI companies have actually made a push to improve their models’ responses to people in crisis.

That’s a reflection of a greater truth about tech platforms, which is that they are more likely to improve their products if they are at risk of being sued.

What people are saying: On X, OpenAI co-founder Wojciech Zaremba complimented Transluce’s work on AI evaluation methodology. Zaremba wrote that current question-and-answer evals don’t cut it as “AI interactions increasingly span days or months,” and “Transluce just pushed this frontier forward with multi-turn evals that use simulated users to measure AI’s effects on mental health.”

—Ella Markianos

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)

(Link)

Talk to us

Send us tips, comments, questions, and safety reports: casey@platformer.news. Read our ethics policy here.