The shape of the problem
An agent modifies a CRM record, sends a customer email, and updates an invoice. The invoice update fails. The framework retries the entire operation. The customer receives the email a second time. The CRM record is modified again, this time overwriting a change made by another agent in the interval between the first attempt and the retry.
This is the default behaviour of any agent execution system that does not enforce idempotency at the substrate level. The agent did nothing wrong. The model reasoned correctly. The framework followed its retry policy. The failure is structural: the system retried an operation that had already produced irreversible side effects.
What makes an operation non-idempotent?
An idempotent operation produces the same result whether it executes once or multiple times. A database read is idempotent. Setting a field to a specific value is idempotent. An HTTP GET request is, by specification, idempotent.
A non-idempotent operation produces additional effects on each execution. Sending an email is non-idempotent: sending it twice means the recipient receives two emails. Incrementing a counter is non-idempotent: incrementing twice produces a different result than incrementing once. Triggering a webhook is non-idempotent: the downstream system processes each invocation independently.
Most agent operations in production environments are non-idempotent, because most useful agent operations involve side effects. An agent that only reads data and returns a summary is idempotent, but it is also limited. The value of production agents comes from their ability to act: modify records, send communications, trigger workflows, update external systems. Every one of these actions is a side effect. Every one of these side effects makes the operation non-idempotent unless specific countermeasures are in place.
The countermeasure is an idempotency key: a unique identifier attached to each operation that allows the substrate to detect and suppress duplicate executions. When an operation is retried, the substrate checks whether an operation with that key has already completed. If it has, the substrate returns the stored result without re-executing the operation. The side effects happen exactly once.
In the SDK, the operation runs through the venue, which is its system of record:
from covia import Grid
venue = Grid.connect("https://venue.covia.ai")
# A side-effecting operation runs on the venue. Because the venue records the
# job, a retry after a crash resumes from the recorded state instead of
# re-emitting the side effect.
result = venue.run(
"billing/charge-customer",
{"customer_id": "cus_123", "amount_cents": 4900},
timeout=30,
)How does retry duplication manifest in production?
The CRM modification pattern
Agent A reads a customer record, decides the customer should be upgraded to a premium tier, and writes the upgrade. The write succeeds, but the subsequent step (notifying the billing system) times out. The framework retries the entire task. Agent A reads the customer record again. It has already been upgraded. Depending on the agent's logic, one of two things happens.
If the agent checks whether the upgrade has already been applied, it skips the write and proceeds. This is the optimistic case, and it requires the agent to be written with awareness of its own retry context. Most agent frameworks do not provide this context. The agent does not know it is retrying. It simply receives a task and executes it.
If the agent does not check, it writes the upgrade again. In some systems, this is a no-op (setting a field to its current value). In others, the write triggers downstream effects: an audit log entry, a webhook to the billing system, a notification to the customer success team. Each of these downstream effects fires again on the duplicate write, regardless of whether the record itself changed.
The email delivery pattern
Agent B composes and sends a status update email to a client. The send succeeds. The agent then attempts to log the sent email in the CRM. The CRM API returns a 503. The framework retries the task from the beginning. Agent B composes and sends the email again. The client receives two identical emails.
This pattern is particularly difficult to address at the application layer because email delivery is a fire-and-forget operation. Once the email is in the SMTP pipeline, it cannot be recalled. The only prevention is to never send it in the first place on a retry, which requires the substrate to know that the send step already completed.
The deployment trigger pattern
Agent C determines that a configuration change requires a redeployment. It triggers a deployment pipeline via API call. The pipeline starts. The agent's next step (updating the deployment record) fails due to a network partition. The framework retries. Agent C triggers the deployment pipeline again. Two deployments run concurrently on the same service, each believing it is the authoritative deployment.
This failure mode compounds: the two concurrent deployments may interfere with each other, producing a state that neither deployment intended. The service may end up running a hybrid configuration that was never tested, never approved, and cannot be reproduced.
Why can application-level deduplication not solve this?
The standard engineering response to duplicate side effects is application-level deduplication. The agent checks whether the action has already been performed before performing it again. This approach has three structural weaknesses.
First, it requires every agent to implement its own deduplication logic. Each agent must track which steps of its workflow have completed, persist that tracking across retries, and consult the tracking before each step. This is substantial engineering work that must be repeated for every agent, and a single omission produces a duplication bug that may not surface for weeks.
Second, it requires the agent to have reliable state across retries. If the agent's own state is lost when it crashes (which is the common case, since most agent frameworks run agents as stateless functions), it cannot check whether a step has already completed. It must query external systems to infer its own execution history, which is fragile and incomplete.
Third, it does not address the window between "action completed" and "completion recorded." If the agent sends an email and then crashes before recording that the email was sent, the next retry will find no record of the send and will send again. This window is irreducible at the application layer. It can only be closed by an atomic substrate operation that executes the action and records its completion in a single, indivisible step.
What is the substrate-level solution?
The substrate-level solution has three components: idempotency keys, step-level execution tracking, and atomic side-effect emission.
Idempotency keys are unique identifiers assigned to each operation at the substrate level, before the agent begins execution. The key is deterministic: given the same task, the same agent, and the same input, the substrate generates the same key. When the substrate encounters a retry, it detects the duplicate key and short-circuits to the stored result.
Step-level execution tracking means the substrate records the completion of each step within a multi-step operation, not just the operation as a whole. When a retry occurs, the substrate knows that steps 1 through 3 completed and step 4 failed. It resumes from step 4 rather than re-executing from step 1. The completed steps are not re-executed, so their side effects are not duplicated.
Atomic side-effect emission means that when a step produces a side effect (sending an email, triggering a webhook, writing to an external system), the substrate records the emission and the step completion as a single atomic operation. There is no window between "side effect emitted" and "emission recorded." If the atomic operation fails, neither the emission nor the recording happens, and the retry executes the step cleanly.
The Grid implements all three of these mechanisms at the substrate level. Agents running on the Grid do not need to implement their own deduplication logic. The substrate handles it, in the same way that TCP handles packet retransmission without requiring the application layer to deduplicate received data.
How does this relate to the guarantees?
Deterministic retry and rollback is the third of seven guarantees that production agent substrates must provide. It interacts directly with three of the others.
Ordered execution ensures that retried operations are sequenced correctly relative to concurrent operations by other agents. Without ordering, a retry could interleave with another agent's modifications in a way that produces an inconsistent state, even if the retry itself is deduplicated.
The canonical system of record ensures that step-level execution tracking persists reliably. If the execution tracking itself is stored in a system that can lose data, the guarantees collapse. The system of record must be at least as reliable as the operations it tracks.
Convergent state ensures that the results of deduplicated operations merge correctly across distributed nodes. If Agent A's operation completes on Node 1 and the retry arrives at Node 2, convergent state ensures that both nodes agree on the final result without requiring Node 2 to query Node 1 synchronously.
The economic argument
Duplicate side effects have direct economic costs. Duplicate emails erode customer trust and trigger spam complaints. Duplicate CRM modifications corrupt sales data and pipeline forecasting. Duplicate deployment triggers waste compute resources and risk service outages. Duplicate financial transactions create accounting discrepancies that require manual reconciliation.
These costs are often invisible in development and staging, because test environments rarely have the concurrency, failure rates, and side-effect consequences of production. A team that tests an agent by running it once on a clean dataset will never observe a retry duplication bug. The bug only manifests under production conditions: concurrent load, intermittent network failures, external API rate limiting, and partial completion of multi-step workflows.
The economic argument for substrate-level idempotency is that the alternative is accepting that every agent operation in production has a non-zero probability of duplicating its side effects on every retry, with costs that compound across every agent and every operation, continuously.
What about operations that are inherently non-idempotent?
Some operations cannot be made idempotent even with substrate-level support. A physical robot picking up an object cannot "un-pick" it. A financial transfer that has already settled cannot be deduplicated after the fact. A published social media post cannot be unseen.
For these operations, the substrate must provide a different guarantee: execute-at-most-once semantics. Rather than retrying the operation, the substrate records the failure and escalates to a governed escalation path. A human or a supervisory agent reviews the failed state and decides whether to retry, compensate, or abort.
Execute-at-most-once is more conservative than execute-exactly-once. It accepts that some operations may fail permanently rather than risk duplication. For operations with high side-effect costs (financial transactions, legal communications, irreversible physical actions), this is the correct trade-off. Better to fail and escalate than to succeed twice.
The substrate's responsibility is to make this trade-off explicit and configurable. An organisation that deploys agents to send marketing emails may accept execute-at-least-once semantics (a duplicate email is annoying but not catastrophic). An organisation that deploys agents to execute financial transactions requires execute-at-most-once semantics (a duplicate transaction is a regulatory incident). The substrate must support both, and the choice must be a configuration decision rather than an architectural constraint.
The deeper question
Idempotency in agent systems is a specific instance of a broader problem: the interaction between probabilistic reasoning and deterministic execution. Agents reason probabilistically. They make decisions based on context, heuristics, and model outputs that are inherently non-deterministic. But the systems they act upon are deterministic. A database commits or it does not. An email is sent or it is not. A deployment triggers or it does not.
The substrate sits at the boundary between these two domains. It accepts non-deterministic instructions from the reasoning layer and translates them into deterministic operations on external systems. The idempotency guarantee is one expression of this translation: ensuring that the deterministic execution layer produces exactly the intended effects, regardless of how many times the non-deterministic reasoning layer requests them.
Whether this boundary is the right place to enforce determinism, or whether the reasoning layer itself should become more deterministic, is a question that the field has not yet settled. The practical answer, for now, is that production systems cannot wait for probabilistic reasoning to become reliable. They need a deterministic substrate today, and the agents can improve on their own schedule.