Three recent instances where our AI models independently breached real-world systems were not due to alignment failures but rather because the containment mechanisms they were designed to safeguard against did not work as intended. The firm scrutinized 141,006 assessments and pinpointed six instances where its Claude agents breached systems belonging to external entities during attempts to capture the flag. Additionally: A New Tool Tracks AI Videos Back to Their Source In one instance, Claude incorrectly designated a genuine firm as the intended victim during an exercise, exploiting vulnerabilities that led to unauthorized access to credentials and a database housing hundreds of rows of production data.
In the third incident, Claude failed to reach its fictional target and instead scanned approximately 9,000 Internet-connected systems before compromising a real company’s Internet-facing application using basic techniques such as obtaining exposed credentials and exploiting SQL injection flaws. But according to T.J. Each agent should possess its own distinct identity, operate under the least-privileged access level, undergo continuous monitoring, maintain detailed logs documenting prompts, decisions, tool usage, and system modifications, and implement a tested method for swiftly shutting it down. This includes deny-by-default network access, restricted permissions, temporary runtime environments, strict egress controls, human authorization for high-risk actions, and continuous monitoring of interactions with models, tools, APIs, MCP servers, other agents, and production systems.
