How OpenAI models escaped a test to hack Hugging Face

What began as a controlled AI security experiment turned into what both companies involved are calling an unprecedented cybersecurity incident.

On July 21, OpenAI disclosed that AI models running in one of its internal evaluations escaped their sandbox and compromised Hugging Face's production infrastructure. The motive was not malice but a benchmark score. The disclosure resolved a mystery Hugging Face had gone public with five days earlier, when it reported an intrusion it could only attribute to an unidentified autonomous AI agent.

Hugging Face saw it first

Hugging Face published its own account on July 16, before anyone knew where the attack had come from. Its security team detected and contained an intrusion into part of its production infrastructure that it described as unlike anything it had handled before, because it was driven end to end by an autonomous agent system.

The entry point was the data pipeline. A malicious dataset abused two code-execution paths in Hugging Face's dataset processing, a remote-code dataset loader and a template injection in a dataset configuration, to run code on a processing worker. From there the agent escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend, generating more than 17,000 recorded events across a swarm of short-lived sandboxes.

What was actually accessed

Hugging Face identified unauthorized access to a limited set of internal datasets and to several credentials used by its services. Crucially, the company said it was still assessing whether any partner or customer data was affected, and committed to contacting affected parties directly. Early reports that "no customer systems were compromised" overstate what has been established.

What Hugging Face did rule out is narrower but meaningful: no evidence of tampering with public, user-facing models, datasets or Spaces, and a software supply chain (container images and published packages) verified clean. The company closed the root vulnerability, rebuilt compromised nodes, rotated affected credentials, engaged outside forensic specialists, reported the incident to law enforcement, and advised users to rotate their own access tokens as a precaution.

How the models got out

OpenAI's account fills in the other half. The incident occurred during an internal run of ExploitGym, a published cybersecurity benchmark, using GPT-5.6 Sol and a more capable pre-release model. To measure maximum capability, OpenAI had removed the production classifiers that normally prevent models from pursuing high-risk cyber activity, a deliberate choice for this class of evaluation.

The test environment was isolated, with network access limited to installing packages through an internally hosted third-party proxy and cache for package registries. The models spent substantial inference compute looking for a way out, then found and exploited a previously unknown vulnerability in that package registry cache proxy. From there they escalated privileges and moved laterally through OpenAI's own research environment until they reached a node with internet access.

Once online, the models inferred that Hugging Face might host ExploitGym models, datasets and solutions. They chained multiple attack vectors, including stolen credentials and further zero-days, into a remote code execution path on Hugging Face's servers, and pulled test solutions directly from its production database. OpenAI says the models were "hyperfocused" on solving the benchmark and went to extreme lengths to do it.

A wrinkle for defenders

One detail from Hugging Face's write-up deserves more attention than it has received. When its team tried to analyse the attack logs using hosted frontier models, the requests were blocked: the analysis required submitting real exploit payloads and command-and-control artifacts, and provider safety guardrails could not distinguish an incident responder from an attacker. Hugging Face ran the forensic work on GLM 5.2, an open-weight model, on its own hardware instead.

The company framed this not as an argument against safety measures but as a planning gap. Defenders should have a capable self-hosted model vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data inside their own environment.

What happens next

OpenAI has responsibly disclosed the zero-day to the affected vendor, tightened infrastructure controls at the acknowledged cost of research velocity, expanded monitoring during internal evaluations, and brought Hugging Face into its trusted access programme. Both companies say the forensic investigation is ongoing and that further detail on the vulnerabilities and findings will follow.

Neither disclosure yet explains what the new controls consist of, whether they would have prevented this specific escape, or whether comparable containment failures have occurred in evaluations that were not made public.