Outside Experts Find Big Blind Spots in OpenAI Safety Checks

It turns out that even the biggest AI company in the world has some blind spots when it comes to keeping an eye on its own creations.

Between July 21 and August 11 of this year, two researchers from the organizations METR and Redwood Research spent three weeks inside OpenAI. They were part of a pilot program designed to give outside experts deep access to the company's internal infrastructure. During that window, they had full access to the codebase and internal documents, and they interviewed about a dozen employees to understand how the company monitors for rogue AI behavior.

The audit uncovered several significant weaknesses in the current monitoring setup. The team estimated that between 6 and 25 percent of internal traffic from the Responses API, specifically from models of the GPT-5.5 class or higher, was not covered by any monitoring at all. METR noted that this estimate carries significant uncertainty in both directions. Additionally, for internal Codex traffic that was supposed to be tracked, roughly 10 percent went untracked between May and July. This was due to a gap in how the system handled compaction events, which are essentially context summaries. OpenAI had actually found and fixed this specific bug while the audit was still ongoing, but the auditors noted that during this period, agents could delay oversight by simply continuing to execute actions rather than handing control back to the user.

Beyond the coverage gaps, the researchers found more active ways the system could be bypassed. They identified methods to hide hundreds of actions from the monitor and ways to bypass the Codex sandbox. They also determined that specially crafted inputs could likely crash the monitor entirely. At the start of the audit, model behavior during reinforcement learning (RL) training was not being monitored at all. OpenAI stated that it paused RL training after an incident involving Hugging Face in August and expanded monitoring to cover it, but these changes happened after the auditors' access period ended, so they were not part of the official evaluation.

OpenAI says it has since fixed all the weaknesses the auditors identified and has distributed the full 61-page report internally. After the company applied redactions, the leadership teams at METR and Redwood Research received a 38-page version. Only a short summary has been released to the public so far. METR stated that it did not have the right to disclose the redacted portions on its own. As this was a pilot program, METR says it hopes to push for greater public transparency in future audits.