The substrate problem
Production AI agents fail in ways that look nothing like demo failures. A demo agent summarises a document, generates a response, and returns it to a user. A production agent modifies a CRM record, triggers a deployment pipeline, sends a customer email, and updates an invoice, all within a single workflow that may involve three other agents operating on the same data concurrently.
The difference is consequence. When a demo agent fails, a user clicks "retry." When a production agent fails mid-workflow, a customer receives a duplicate invoice, a deployment rolls forward on corrupted state, or two agents overwrite each other's modifications to a shared record.
Most teams discover this the hard way. They build agents that work reliably in isolation, deploy them into shared production environments, and watch failures cascade in patterns they did not anticipate. The agents themselves are fine. The substrate they run on provides no guarantees about what happens when multiple agents act concurrently on shared systems.
These are substrate guarantees, not application features. You cannot bolt them on at the application layer any more than you can implement TCP reliability in a web application without a transport protocol underneath. The agent framework does not matter. The model does not matter. What matters is whether the execution substrate enforces the properties that production workloads require.
What qualifies as a production agent?
A production agent is any autonomous process that modifies shared state or triggers side effects in systems where other agents, humans, or processes also operate. The defining characteristic is participation in a shared environment where actions have consequences that persist beyond the agent's own execution context.
A production agent is distinct from a tool-augmented chatbot in three ways. First, it acts without waiting for human approval on every step. Second, its actions affect systems that other agents also modify. Third, its failures have consequences that cannot be undone by refreshing a browser window.
By this definition, most "AI agents" in production today are production agents whether their operators realise it or not. The moment an agent writes to a database, sends a message, or calls an external API, it is operating in production territory. The question is whether the substrate it runs on acknowledges this.
The seven guarantees
1. Ordered execution
When Agent A reads a customer record, decides to update the billing tier, and writes the new tier back, the read and write must execute in a defined order relative to every other operation on that record. If Agent B is simultaneously modifying the same customer's support tier, the final state must reflect a serialisable ordering of both operations.
Without ordered execution, the outcome depends on timing. Two agents reading the same record at the same instant, each making a decision based on that read, each writing back a modification: the last write wins, and one agent's work is silently discarded. The agent that lost the race has no way to detect this. It completed successfully from its own perspective.
Ordered execution does not mean sequential execution. Agents can and should run concurrently. Ordered execution means the substrate maintains a consistent ordering of state transitions, so that every agent's view of shared state reflects a coherent history. This is the same guarantee that databases provide through serialisable isolation levels, applied to the broader surface area of agent operations.
2. Scoped authority
Every agent action must execute within a defined permission boundary. Scoped authority means an agent can modify the resources it has been granted access to and nothing else, enforced by the substrate rather than by the agent's own restraint.
A scoped authority system defines, for each agent, which resources it can read, which it can modify, and which operations it can invoke. These scopes are enforced at the substrate level. An agent that attempts to exceed its authority receives an explicit denial.
This matters because agents make decisions based on context, and context can be manipulated. A prompt injection that convinces an agent to "update all customer records" should fail because the agent's execution scope limits it to the specific customer records relevant to its current task. The enforcement boundary must sit below the reasoning layer, in the same way that operating system permissions sit below the application layer.
Scoped authority also enables auditability. When every action is tagged with the agent's identity and scope, the system of record can answer questions like "which agent modified this record, and did it have permission to do so?" without relying on the agent to self-report honestly.
3. Deterministic retry and rollback
When an agent operation fails mid-execution, the substrate must provide exactly two options: retry the operation from a known-good state, or roll back all of its effects. "Retry from wherever it crashed" is not an option, because the operation may have already produced side effects that cannot be duplicated.
A deterministic retry means the substrate knows exactly which steps completed, which side effects were emitted, and which steps remain. It replays only the incomplete portion, using idempotency keys to prevent duplicate side effects. A deterministic rollback means the substrate can reverse all effects of the failed operation, returning shared state to its pre-operation condition.
Without this guarantee, retries in non-idempotent tasks duplicate side effects. An agent that sends a customer email, then fails on the next step, will send that email again on retry unless the substrate deduplicates it. An agent that debits an account and then fails will debit twice. The failure mode is that the agent crashed and the recovery path made it worse.
4. Persistent memory, execution-linked
Agent memory must persist across executions and must be linked to the execution context that produced it. This is distinct from both conversation history (which is session-scoped) and vector databases (which are query-scoped).
Execution-linked memory means that when an agent runs a workflow on Monday and then continues that workflow on Wednesday, it has access to the complete state of Monday's execution: which steps completed, what decisions were made, what intermediate results were produced. This memory is the canonical record of what happened, stored in the same substrate that enforced the execution guarantees.
This matters for long-running workflows. An agent coordinating a multi-week procurement process needs to remember not just "what was the last thing I did" but "what is the complete state of every sub-task, which approvals are pending, and what decisions did I make three weeks ago that constrain my options today." Persistent, execution-linked memory makes this a substrate feature rather than an application-level reconstruction problem.
What does a system of record mean for agent execution?
5. Canonical system of record
A system of record for agent execution is a single, authoritative source of truth for every action taken, every state change produced, and every decision made by every agent in the system. It is not a log. Logs are append-only records that require reconstruction to answer questions about current state. A system of record maintains the current state directly, alongside the full history that produced it.
The distinction matters operationally. When an incident occurs, the first question is "what happened?" A log-based system requires an engineer to reconstruct the sequence of events from potentially millions of entries across multiple services. A system of record answers the question directly: here is the current state, here is every transition that produced it, here is which agent made each transition and under what authority.
The Grid implements this as a first-class substrate feature. Every agent operation, every state transition, every authority check, and every retry or rollback is recorded in a single, queryable system of record that provides both the current state and the complete causal history.
6. Governed escalation
Not every decision should be made by an agent. Governed escalation means the substrate provides a formal mechanism for agents to escalate decisions to humans, other agents, or governance policies, with the escalation itself recorded in the system of record.
Governed escalation is distinct from "human in the loop." Human-in-the-loop implies a binary: either the human approves every action, or the agent acts autonomously. Governed escalation defines a spectrum. The substrate enforces policies like "this agent can approve expenses under $1,000 autonomously, must request peer review for $1,000 to $10,000, and must escalate to a human for anything above $10,000."
The policies are enforced at the substrate level. An agent that encounters a decision exceeding its authority does not choose to escalate. The substrate prevents it from proceeding and routes the decision to the appropriate authority. The escalation, the decision, and the resumption of the original workflow are all recorded in the system of record.
7. Convergent state
When multiple agents operate across distributed systems, their local views of shared state will temporarily diverge. Convergent state means the substrate guarantees that all divergent views will eventually resolve to the same value, without requiring a central coordinator or leader election.
A convergent state system uses data structures with mathematically guaranteed merge properties. When two agents modify the same data concurrently, the merge function produces the same result regardless of the order in which the modifications are applied. This eliminates an entire class of distributed systems failures: split-brain scenarios, stale reads that produce conflicting writes, and coordination deadlocks.
Convergent state does not mean eventual consistency in the traditional sense. Eventual consistency often means "the system will probably converge at some point, and you can read stale data in the meantime." Convergent state means the data structures themselves guarantee convergence, with deterministic merge semantics that produce a provably correct result.
The compound effect
These guarantees are not independent features. They form a compound property of the execution substrate. Ordered execution without scoped authority means operations are sequenced correctly but any agent can modify anything. Scoped authority without a system of record means permissions are enforced but no one can verify after the fact. Deterministic retry without convergent state means individual operations recover correctly but the global state can still diverge.
The compound effect is this: a substrate that provides all seven guarantees simultaneously creates an environment where production agents can operate concurrently on shared systems with the same reliability expectations as traditional distributed services. The substrate enforces properties that the agents cannot violate.
This is the argument for treating the coordination layer as infrastructure rather than as a feature of individual agent frameworks. Each framework can implement some of these guarantees for its own agents. No framework can enforce them across agents from different frameworks operating on the same shared state, unless the enforcement happens at the substrate level.
What remains unsolved
These guarantees address the substrate layer. They do not address the reasoning layer. An agent with perfect execution guarantees can still make bad decisions. It can correctly and reliably execute a plan that is wrong. The substrate ensures the execution is faithful; it does not ensure the plan is sound.
This is the boundary between infrastructure and intelligence, and it is a boundary worth respecting. The history of computing infrastructure shows that conflating these layers produces systems that are fragile at both. Operating systems do not evaluate the quality of the programs they run. Databases do not assess the wisdom of the queries they execute. The execution substrate for agents should enforce execution guarantees without opining on execution intent.
The open question is where reasoning governance belongs. If the substrate cannot evaluate intent, and the agent framework trusts the model's judgment, and the model is probabilistic by nature, who is responsible when a correctly executed plan produces a harmful outcome? The guarantees ensure the execution is faithful. Faithful execution of a flawed plan is a failure mode that no substrate can prevent.