Idempotency in Workflows: Preventing Duplicate Runs and Side Effects

By Chris Moen • Published 2026-04-17

Learn how to prevent duplicate workflow runs and unwanted side effects by implementing idempotency patterns. This guide covers practical strategies like using unique keys, deduplication stores, and deterministic IDs.

Quick answer

Prevent duplicate workflow runs by making retries idempotent. Use a unique key per event or request, check it before any mutation, and return the prior result on duplicates. Pair this with safe retry rules and tight concurrency.

What idempotency means in practice

Idempotency means that running the same request more than once gives the same effect. Side effects do not double. It works by tying each request to a stable key. The system checks that key. If it has been handled, it does not run the mutation again. It can also return the previous result.

Many webhook providers use at-least-once delivery, so duplicates happen in normal operation. See a practical note from the n8n community on this retry behavior and why you should gate mutations with a key check (Preventing duplicate webhook executions).

Why this matters for production workflows

Retries can double-charge cards or create duplicate orders.
Human double-clicks and page refreshes can fire the same action twice.
Network timeouts trigger client or provider retries.
Concurrency can race two runs at the same time.

Idempotency turns these from outages into non-events. The second run is a no-op or a cache hit.

Where duplicates come from

At-least-once webhooks and queue retries
Client-side retries and refreshes
Scheduled jobs that overlap
Slow APIs and timeouts that trigger a retry layer
Concurrency on hot endpoints or shared workers

Core patterns that stop duplicates

Use grouped patterns. Do not rely on a single guard.

Request and event keys
Generate or accept an idempotency key.
Store the first successful result with that key.
On duplicate, skip the mutation and return the stored result.
This mirrors the cookbook pattern to accept a key and return a cached result on duplicates (idempotency keys).

Dedupe store with check-before-mutation
Keep a small store of processed keys.
Do the check before any side effect.
Use an atomic set-if-absent to avoid race windows. Community threads note Redis SET NX as a simple way to get both the check and lock in one call (discussion).

Deterministic workflow IDs
Derive workflow run IDs from the trigger event ID.
Reject or no-op if a run with that derived ID already exists.
Useful for webhooks and scheduled imports.

Upserts and unique constraints
Use upsert, not blind insert, for orders, payments, and emails.
Add unique indexes on external IDs and transaction IDs.
Let the database enforce uniqueness and handle duplicates cleanly.

Sagas and compensations
Split long flows into steps with a rollback path for each step.
If a retry runs partway, use the recorded state to skip or compensate.
This pattern is common in distributed workflow guides, including agent and tool design overviews (patterns overview).

Outbox and delivery logs
Write events to an outbox table first.
Publish from the outbox and mark sent with a unique delivery key.
Consumers use inbox dedupe or idempotency keys to avoid replay effects.

Backoff and bounded retries
Use exponential backoff and a cap on total attempts.
Stop retry storms and space out load.
Many reliability guides pair retry policy and idempotency as a set (retry and backoff guidance).

Durable execution and result caching
Persist step outputs and status.
On retry, skip completed steps and reuse results.
This is a common theme in distributed workflow posts, such as Orkes guidance on retry-safe design (retry-safe workflows).

How Breyta fits this use case

Breyta is a workflow and agent orchestration platform for coding agents. It gives you structure, state, and clear run history around your flows. That makes idempotent retries practical to implement and verify.

What helps in Breyta:

Deterministic orchestration
Flows run with clear, step-by-step behavior.
You can see exactly where a retry would resume or skip.

Triggers and versioned releases
Webhook, manual, and schedule triggers are supported.
Draft vs live split gives safe rollout and pinned releases per run.

Waits, approvals, and callbacks
Pause with a wait.
Resume on external callbacks.
Keep a human approval gate where side effects are sensitive.

Long-running agents
Kick off remote or VM-backed agents over SSH.
Use a wait step and resume when the worker posts to a callback.
This pattern avoids holding open fragile steps during retries.

State and resources
Treat large outputs as resources with refs.
Persist artifacts without bloating step state.
Reuse outputs across retries without recompute.

CLI and agent-first operation
The CLI returns stable JSON.
Agents and scripts can check run history, resources, and keys.
This makes automation around idempotency checks reliable.

Implementing idempotency in a Breyta flow

Here is a simple pattern for webhooks and other at-least-once triggers.

Compute a dedupe key
Derive it from the provider’s event ID or a hash of the request body.
Normalize fields that vary.

Do a key check before any mutation
Use a small store with Breyta step families like :kv or :db.
Attempt an atomic write for the key.
If the key exists, return early with a 200 OK and a note that it was handled.

Record the side effect result
Run downstream actions.
Persist a compact result or a resource ref.
Update the key entry with status and a link to the run.

Make the response deterministic
On duplicates, return the same status and the prior result ref.
This keeps callers simple and safe to retry.

Example step outline:

1) Trigger: webhook 2) :function → build dedupe key from payload 3) :kv → set-if-absent(key). If absent, continue. If present, short-circuit 4) :http or :db → perform side effect (charge, write, send) 5) :persist → store large output as resource, save res:// ref 6) :notify or :http → return result and ref 7) On error → keep the key but mark as failed or expiring, then retry safely

Tips:

Use an approval step before money movement.
Set a short TTL for keys when the action is safe to retry later.
Keep a run link in the key value for fast debugging.

Handling long-running and agent-driven work

For VM or local agents:

Start remote work over :ssh.
Add a :wait with a callback URL that includes the same job key.
On callback, recheck the key and resume.
Persist the final output as a resource ref.

Why this helps:

You keep state across time.
Retries do not reopen the same long SSH step.
The callback uses the same idempotency key, so the finish step is safe even on duplicates.

Monitoring and safe rollout

Use clear run history to verify that duplicates short-circuit early.
Group logs by dedupe key for quick checks.
Promote from draft to live only after duplicate tests pass.
Pin runs to the resolved live release. This avoids odd shifts during upgrades.
Keep notifications on for blocked duplicates during the first weeks.

FAQ

Is deduplication the same as idempotency?

They are related but not the same. Deduplication drops repeats. Idempotency makes repeats safe and returns the same outcome. You often use both.

What if a step cannot be idempotent?

Add a key gate before it. Consider an approval step. If possible, switch to an upsert or a safe read-modify-write. If not possible, use a saga to compensate on failure.

Do I still need a database check if I use keys?

Yes. Keys stop re-entry. Unique constraints and upserts protect the data layer. Use both for defense in depth.

Summary

Make retries safe by default. Use a key per event, check before mutation, and cache results. Breyta’s structured flows, waits, retries, and timeouts, versioned releases, and clear run history help you build and operate these patterns with confidence.