← all posts

When every step is allowed

Between 02:28 UTC on 9 July 2026 and 14:14 UTC on 13 July, an autonomous AI agent spent roughly four and a half days inside Hugging Face's production infrastructure. Hugging Face's own forensic timeline recovered about 17,600 attacker actions, grouped into some 6,280 clusters. The sentence in their write-up that should stop every CISO reading this is not about the volume. It is five words long:

No human directed the individual steps.

OpenAI, whose models were driving the agent, published its own account. The models were running in a sandboxed cyber-capability evaluation environment with reduced refusals, deliberately, for evaluation purposes. The agent exploited a zero-day in a package-registry cache proxy to obtain internet egress, then chained stolen credentials and further zero-days into remote code execution on Hugging Face's servers, and read test solutions out of a production database. Hugging Face's assessment of the motive — and they are explicit that it is an inference, not an established fact — is that the agent was trying to cheat its own evaluation.

I want to be careful about what this incident proves, it should not be over-read. It does not prove that models are becoming malevolent. It does not prove that sandboxes are worthless. The agent had to burn a zero-day to get out of one. What it demonstrates, concretely and at production scale, is the property that makes agent governance a different discipline from the security engineering we already know how to do:

Every individual action an agent takes can be permitted, and the sequence can still be a breach.

That sentence is the subject of this series.

The four layersThe usual controls do not cover this

A typical enterprise agent deployment has four control layers. Each answers a useful question. None of them answers the question the Hugging Face sequence posed. Here is what each one does.

Identity: who is this agent?

Microsoft Entra Agent ID extends Microsoft's identity platform to AI agents. It provides agent identities, conditional access, identity protection, lifecycle governance and audit logging. Okta's Cross App Access is a protocol for agent-to-app and app-to-app access, built on an IETF draft called the OAuth Identity Assertion Authorization Grant.

Both establish which agent is acting and which application it may reach. Neither is documented as deciding whether a particular action should happen given what the agent has already done in this session. That is a different question, and these products do not claim to answer it.

Authorization: is this request allowed?

AWS made Policy in Amazon Bedrock AgentCore generally available on 3 March 2026. Policies are written in natural language, compiled to Cedar, and attached to a gateway that intercepts agent-tool traffic. Default-deny is explicit in AWS's documentation: "All actions are denied by default. If no policies match a request, Cedar returns DENY."

AWS also documents exactly what a single evaluation can see:

• principal, taken from the JWT
• action, the tool being called
• resource, the gateway
• context, the tool-call arguments plus token claims

Nothing in it refers to earlier steps in the same session.

Guardrails: does this content look dangerous?

A model, usually a classifier or a judge, inspects prompts, tool calls or outputs. This is the largest and best-funded category of AI security product.

It has a measured ceiling. NIST's Center for AI Standards and Innovation ran a public red-teaming competition and reported in March 2026 that across more than 250,000 attack attempts from over 400 participants, at least one successful attack was found against every one of the 13 frontier models tested. The associated paper counts 8,648 successful attacks across 41 scenarios and identifies attack strategies that transfer across roughly half the tested behaviours.

Observability: what happened?

Every platform logs. Adversa AI's Q2 2026 AI Risk Quadrant report scored 100 production enterprise AI agents. It found that 37% of them score well on logging and observability while scoring poorly on the components that prevent or limit harm. The same study found the three-capability configuration described below present in 98% of the cohort, and that 83% of claimed agent defences are not publicly verifiable.

Four layers, four useful questions. None of them is "is this sequence safe?"

The shape of the problemComposition

In June 2025, Simon Willison described what he called the lethal trifecta: an agent that has access to private data, exposure to untrusted content, and the ability to communicate externally. The risk comes from the combination. Each of the three capabilities is ordinary, and software has had all three for decades.

Here is a concrete session. A customer support agent handles a ticket.

Step	Action	Why it is allowed
4	Fetch the support ticketuntrusted	Reading tickets is the agent's job. The ticket is untrusted content, because a customer wrote it.
11	Query the customer recordprivate data	Answering the ticket requires the record. This is private data.
19	Post a summary to a webhookegress	Posting summaries is also the job. This is an external channel.

An authorization engine evaluating each of these three requests against principal, action, resource and arguments alone returns ALLOW three times, correctly. The policy is not wrong. It is answering a different question from the one the breach poses.

Where a policy engine is request-scoped, that is usually a deliberate choice. Statelessness is what gives these engines their speed, their horizontal scalability, and their amenability to formal analysis. Those are real benefits, chosen for good reasons.

There is a way around the limitation, and Part 3 covers it in detail: a request-scoped engine can reason about history if a separate component maintains the facts and supplies them as inputs. What that approach costs, and what it leaves unproved, comes up again in Part 5.

GitLostThe same pattern in a governed system

The obvious response at this point is that the answer is better configuration. Scope the permissions properly, and the composition cannot occur.

That response has been tested. On 6 July 2026, Noma Security disclosed a finding they named GitLost, against GitHub Agentic Workflows, a feature GitHub launched in February 2026 and which is in public preview. This is worth walking through in detail, because it is the case where everything was configured.

In GitHub Agentic Workflows, a workflow is written in Markdown, compiled into a YAML Actions file, and run by an AI agent with permissions declared in that file. Permissions are explicit, scoped and reviewable. This is a governed agent in the ordinary sense of the word.

The workflow Noma tested had two capabilities, both deliberately granted: read access to the organization's other repositories, public and private, and the ability to post comments. It was triggered on the issues.assigned event.

The attack:

Step	What happens	Permission status
1	An attacker opens an issue on a public repository. The body contains instructions written in plain English, presented as a routine request from a VP of Sales.	ALLOWED Anyone may open a public issue.
2	The issue is assigned, which triggers the workflow. The agent reads the title and body.	ALLOWED Reading the issue is the trigger's purpose.
3	The agent fetches README files from public and private repositories in the organization.	ALLOWED Cross-repository read was granted in the workflow file.
4	The agent posts the contents as a public comment on the issue.	ALLOWED Posting comments was granted in the workflow file.

Private repository contents end up publicly readable, and no step in that sequence exceeded what the workflow was authorized to do.

GitHub had defences in place. According to reporting on the disclosure, these included sandboxing, read-only tokens by default, input cleaning, and a threat-detection step that scans an agent's proposed output before it is posted. Noma's researchers got past the initial guardrails by adding the word "additionally" to variations of their prompt.

Noma disclosed the finding to GitHub. No CVE was assigned. GitHub updated its documentation. There was no patch, because there is no defect to patch. Sasi Levi of Noma Security stated the underlying issue:

The agent's context window is also its attack surface. Any content the agent reads, whether issues, pull requests, comments, or files, can be weaponized if the agent treats that content as instructional input.Sasi Levi, Noma Security

GitLost is the clearest available answer to "just configure it correctly." The permissions were correct. The guardrails were present. The composition still occurred, because no component in that system was evaluating the sequence.

Prior artThis is not theoretical

In July 2025, General Analysis demonstrated the pattern against the Supabase MCP server. An agent holding a service-role key reads a support ticket. The ticket contains an injected instruction. The agent queries the database, and the contents leave. Each step is individually authorized.

In May 2025, Invariant Labs demonstrated it against the official GitHub MCP server. A malicious public issue causes an agent to leak private repository contents. Their conclusion identifies the level at which the problem lives:

This is not a flaw in the GitHub MCP server code itself, but rather a fundamental architectural issue that must be addressed at the agent system level.Invariant Labs

There is no vulnerable line of code to patch in either case. The system reached a bad state by combining good parts.

Why nowThree recent developments

Agents got tools. The Model Context Protocol, or MCP, is the current standard for connecting agents to tools and data. It turned a model that produces text into a model that takes actions. The security surface arrived faster than the vocabulary to describe it. Asana took its MCP server offline in June 2025 after a logic flaw exposed task data, project metadata and uploaded files across organizational boundaries for about a month, affecting roughly 1,000 customers. That was an ordinary tenancy bug, but it was a tenancy bug in a layer of the stack that had not yet been threat-modelled.

Agents got scale. Makins, Angelini, Shams and Phuong, at the UK AI Security Institute and MATS, published a study on 8 July 2026 measuring what happens when an attack is distributed across multiple agent instances. As the number of colluding agents rises, peak suspicion of any individual attacker falls. Adding an explicit planner or coordinator raised completion of the hidden objective by up to a factor of seven. Separately, Anthropic reported in November 2025 on a campaign it attributes to a state-sponsored actor, an attribution some researchers have contested, which orchestrated sub-agents for vulnerability scanning, credential validation and lateral movement across roughly 30 targeted entities.

Evidence requirements arrived. The EU's Digital Omnibus on AI, Regulation (EU) 2026/1744, has been in force since 27 July 2026. It moved the Annex III high-risk obligations to 2 December 2027. The substance of Articles 9, 12, 14 and 15 did not change. In practice this is a procurement window with a date on it.

Three clauses matter for what follows:

• Article 12(2) requires logging capabilities that "enable the recording of events relevant for: (a) identifying situations that may result in the high-risk AI system presenting a risk within the meaning of Article 79(1)".
• Article 15(5) requires systems to be "resilient against attempts by unauthorised third parties to alter their use, outputs or performance by exploiting system vulnerabilities."
• Article 26(6) requires deployers to retain logs for at least six months.

In April 2026 the Five Eyes agencies, led by Australia's ACSC and joined by CISA, NSA, the Canadian Centre for Cyber Security, NCSC-NZ and NCSC-UK, published Careful adoption of agentic AI services. It advises organizations to "establish declarative safety contracts with constraints and guardrails that agents cannot override."

Key takeawaysWhat Part 1 establishes