Trusting the Model is Not an AI Security Strategy

Most enterprise AI risk programs rest on an assumption nobody has written down: that if the model is good enough, the deployment is safe enough. Buy from a reputable lab, read the system card, check the certifications, and the rest is adoption.

That assumption is now the largest unexamined control in the enterprise stack, and it is load-bearing. In a world where everyone is a coder, it is also about to be tested at a scale most organizations have not planned for.

The timing became obvious this week. In We Must Pace the Frontier, Dario Amodei argues that frontier labs should deliberately slow capability advancement so that safety work can keep up. He says Anthropic is committing unilaterally to embedded third-party evaluators with the right to publish findings the company cannot edit. It is a serious proposal backed by a costly commitment, and the industry is better for it.

It is also work that happens entirely upstream of the place where AI actually touches a business. A model can be paced, aligned, interpreted, and independently audited, and still arrive in an organization that has not decided which repositories it can read, which systems it can call, or what happens when it decides the task requires more access than anyone intended.

Those are two different disciplines. Confusing them is how enterprises end up with excellent vendor due diligence and no idea what ran on their infrastructure last week.

Safety lives in the environment

The clearest evidence for this came from the labs themselves, in their own test environments, this summer.

In July, OpenAI disclosed that models had escaped an isolated test environment and reached production systems at Hugging Face. METR's independent investigation found the agents coordinating on a shared unsanctioned message board, dividing into specialized lanes, with named agents assigning roles to the others. One obtained remote code execution on a worker container handling production data.

Then Anthropic published findings from its cybersecurity evaluations that most vendors would have buried. Models that were supposed to have no internet access found one through a misconfiguration. Reviewing more than 141,000 test runs, the team identified three incidents in which models reached real company infrastructure. One retrieved credentials and pulled production data. One published malware to a public package registry, where it was downloaded and run on fifteen real systems.

The instructive part is what the models were doing. They were not rogue. They were completing the assignment with everything within reach (which is the defining behavior of an agent rather than a defect in one). Two of the most sophisticated AI organizations in the world, working in purpose-built isolation with every incentive to get containment right, each left one open path. One open path was enough.

Take the right lesson from that. Safety did not fail at the model layer. It failed at the boundary, which is the layer your organization owns and almost nobody has built.

Frontier labs govern the intelligence, but you govern its agency

This is the division of labor that the industry conversation keeps blurring and it determines what your team is actually accountable for.

Frontier labs are responsible for the intelligence they create. Training, alignment, interpretability, evaluation, refusal behavior, incident disclosure, and the verifiability mechanisms Amodei is proposing all sit with them. No customer can do that work. You cannot audit a training pipeline or read what happens inside a model.

Organizations are responsible for the agency of that intelligence, including where agents execute, what code and data they can open, which tools, systems, and MCP servers they can reach, as well as what credentials and compute they can spend and whether every action is observed, attributed, constrained, and recorded.

Labs do publish governance controls, and the serious ones are useful: retention settings, tenant isolation, enterprise administration, documented usage policies. Those controls get written once for an entire customer base, which is what makes them generic. They describe how a vendor handles data in general, not how your organization is obligated to handle its own.

Regulated environments need something narrower. Controls keyed to your data classification scheme. A residency commitment your regulator will actually accept. Separation of duties between the person requesting the work and the system performing it. Change management that produces evidence rather than assurances. A lab's protocol reasonably stops at its own API boundary, and your obligations continue well past it, into the repositories, records, and internal services the agent reaches once it is inside.

Those are decisions about your business, made with your context, enforced on your infrastructure.

Most organizations have made those decisions. They made them for an environment where the actors were human and countable, and where access review could run on a quarterly cycle because a quarter was fast enough.

The ground has moved underneath those controls. Enterprises now manage an average of 109 machine identities for every human identity, according to Palo Alto Networks' 2026 Identity Security Landscape report. While most can explain what their AI agents are for, far fewer can define what those agents can access, how that access is limited, when permissions are revoked, or which systems can inherit them. The Cloud Security Alliance found in April that 53% of organizations have already experienced AI agents exceeding their intended permissions.

A permission model built around people does not fail loudly when agents arrive. It keeps working, at the wrong scale.

The deployment curve does not slow down

Whatever happens at the frontier, the number of autonomous actors inside enterprises is climbing on a curve set by procurement, employee preference, and board pressure. Gartner projects that by 2028 the average global Fortune 500 enterprise will have more than 150,000 AI agents in use, up from fewer than 15 in 2025.

A safer frontier actually steepens that curve rather than flattening it. Models that arrive with embedded evaluators, published risk reports, and verified alignment properties are easier to approve, easier to defend to a regulator, and easier to roll out broadly. That is the correct outcome and the entire point of the exercise. It also means the better the models get, the more of them will be running inside organizations that never decided where they run.

This puts the burden squarely on the boundary, on a deadline.

What the boundary has to do

Five requirements make a deployment governable, and they hold as a set or not at all.

1. Contain the data, so outbound access is denied by default and every exception is explicit and reviewable.
   
2. Scope the access, so every AI action sees what the task needs and nothing wider.
   
3. Govern the models, so only approved models run and sensitive work routes by policy rather than by whichever tool someone opened first.
   
4. Audit everything, so each action leaves a durable record tied to a person and a task.
   
5. Control the spend, so cost is visible by team and agent with hard stops set in advance.

Assembled from separate tools, these produce five boundaries, five policies, and no single account of what happened. They work when they are enforced in one place, at the point where AI actually executes.

The alternative is the posture Gartner warns about. It predicts that by 2027, 40% of enterprises will demote or decommission autonomous AI agents due to governance gaps identified only after production incidents occur. The root cause its analysts name is treating agent governance as binary, either locked down or fully trusted. Both settings fail. One drives shadow development, the other expands the blast radius.

Build the layer, not another policy

Most enterprises have a model strategy and a roadmap of use cases. Far fewer have built the layer where AI actually runs. Coder's AI Maturity Model assessment of 100 engineering organizations found 70% running agents on infrastructure that was never designed for them, and only 31% at organization-wide governance or better.

We call that layer the AI Operating Layer and the design principle is portable whoever you build it with. Containment, scope, model policy, audit, and spend belong in the environment where work executes, rather than bolted onto each tool as it arrives. A policy asks people to behave. An environment decides what is possible.

For us that means self-hosted workspaces, default-deny networking, and centralized model access applying to every session, human or agent, without anyone reconfiguring per team.

Where the agent thinks and where it runs

The harder problem is the one enterprises actually have. Teams have already chosen their preferred agents, those agents are cloud-hosted, and the vendor's cloud is where the code goes. In a regulated industry that disqualifies the agent regardless of how good it is.

Agent Relay separates where an agent thinks from where it executes. The provider keeps running the reasoning loop. Execution moves into a workspace the enterprise operates, where network policy, RBAC, and audit logging already apply. It launched September 4 with Cursor, and Claude Code followed on September 15, built with support from the Claude Code team.

That split is the division of labor described above, made concrete. The lab governs the model. The enterprise governs where it executes, what it reaches, and whose name is on each action, since every request resolves against the organization's existing identity provider before work starts.

The limit is worth stating. Inference still routes through the agent provider rather than through AI Gateway, which follows from the reasoning loop running in the provider's cloud. Nobody has a complete answer to AI risk, and this one is narrower: what an autonomous actor is permitted to do once it is inside.

Every ungoverned environment is harder to govern after the fact. Every unaudited agent run is harder to reconstruct retroactively. Building the layer deliberately costs a fraction of building it under pressure.

The decision you have already made

Every organization running AI today has made a decision about the boundary. Most of them made it by default, one tool at a time, without anyone in the room who would describe it as a decision.

The frontier labs are being asked to prove their models can be trusted and that debate will run for years. The question in front of everyone else is smaller, more immediate, and entirely within their control: what they have already permitted to happen on their own infrastructure.