Idempotency Stores & the Effect Guarantee

The Idempotency page covers the helper and the consumption contract. This page is the companion deep-dive: the Store behind the helper, the exactly-once-effect guarantee it buys you under at-least-once delivery, the dedup-key contract, and the store backends — the in-memory reference every SDK ships, and how to back it with a shared persistent store for a production fleet.

Status: Authoritative · Helper layer · envelope frozen at schema_version: 1

Nothing here touches the wire. The envelope is frozen at schema_version: 1; meta.id already carries the per-message identity the dedupe is keyed on. The store is a consumer choice about how it treats that field, not a protocol change.

At-least-once delivery → exactly-once effect

Every broker BabelQueue speaks is at-least-once: a message can arrive more than once — a worker crashes after running the handler but before the ack, a broker redelivers on a visibility-timeout lapse, a proxy double-sends, an operator replays a queue. The wire contract makes this explicit and tells handlers to be idempotent (error-handling §1).

The idempotency helper mechanises that: it gives you exactly-once effect without exactly-once delivery. Delivery stays at-least-once — the broker may still hand you the same message many times — but the side-effect (the charge, the email, the row insert) fires once. The store is what makes the difference: it remembers which meta.ids have already produced their effect, so a redelivery is recognised and skipped.

This is not exactly-once delivery (no broker offers that without a transaction the queue cannot see) and not an in-flight concurrency lock — two deliveries of the same id that race before the first completes may both run. It is seen-set, post-success dedupe: the honest, dependency-free guarantee that composes with retry and dead-letter.

The dedup-key contract

The dedupe key is meta.id, verbatim — the canonical, unique identity of this specific message, minted by the first producer and carried unchanged across every hop. It is not trace_id: trace_id spans a whole causal chain, so many distinct messages share one (see id vs trace_id). Keying on trace_id would wrongly collapse unrelated messages; keying on meta.id dedupes exactly the redeliveries of one message.

Because meta.id is part of the frozen envelope, the key is the same in every language: a payment a PHP or Python producer sent is de-duplicated by a Go, Java or Node consumer — same envelope, same identity, one effect. A message with no usable meta.id (empty or absent) cannot be deduped, so the helper fails open: it runs the handler unchanged rather than silently dropping work it cannot identify.

The Store

Every SDK exposes the same three-method record of processed ids, keyed on meta.id:

The helper drives those three methods around your handler:

1. Read meta.id. No usable id → run the handler (fail-open) and return.
2. seen(id) is true → skip: do not invoke the handler; return so the runtime acks it and the broker stops redelivering.
3. Otherwise run the handler; on success only, remember(id). A handler that throws leaves the id unmarked, so a redelivery runs it again — retry / dead-letter still apply. forget(id) later un-remembers an id so a deliberate replay runs once more.

The store answers “was this id processed?” — never “what did it return”. Queue handlers have no response to replay (unlike an HTTP idempotency key), so a seen-set is all the contract needs.

Store backends

The in-memory reference (ships in every core)

Each SDK ships a reference in-memory store implementing the three methods over a process-local set or map. It is the right choice for tests and a single-process consumer, and it is what the conformance fixtures run against.

It is process-local and not persistent: two workers each hold their own set, neither sees the other’s “seen” ids, and a restart forgets everything. That is fine for one process; it is not enough for a fleet.

A shared persistent store

A production fleet needs a shared store so every worker dedupes against the same record — and one that survives a restart. The interface is the extension point: implement the same three methods (seen / remember / forget) over a shared backend and pass your store to the helper unchanged — no other code changes, because the helper only ever calls those three methods. You can still bring your own; Go and PHP now also ship ready-made persistent stores so you usually don’t have to:

The backends follow the obvious shapes behind the same three methods:

• Redis — one key per id with a TTL. seen is EXISTS, remember is SET (a TTL caps unbounded growth), forget is DEL. Fast, shared across workers, naturally expiring.
• A database table — a (message_id PRIMARY KEY, …) row. remember is an INSERT; a unique-constraint conflict is the “already seen” signal. Durable, queryable.

Pick a TTL deliberately. A seen-set grows forever without one. Size the TTL to your broker’s maximum redelivery window — long enough that no legitimate redelivery outlives the remembered id, short enough to bound storage. forget is the manual escape hatch for a planned replay. The shipped stores expose a TTL option (WithTTL in Go; a TTL on the claim in PHP).

Closing the in-flight race: the claim contract

The base seen-set is post-success dedupe: two deliveries of the same id that race before the first completes may both run (it is not an in-flight lock — see the Idempotency page). The shipped persistent stores add an atomic claim to close that window for callers who want it:

• Go — postgres.Store and redis.Store expose Claim(ctx, id) (bool, error): an atomic INSERT … ON CONFLICT DO NOTHING (Postgres) or SET … NX PX (Redis). The winner gets true; a concurrent duplicate gets false and parks.
• PHP — the BabelQueue\Idempotency\ClaimingStore interface extends IdempotencyStore with claim($id, $ttlSeconds): bool and release($id): void; PdoStore and RedisStore implement it. claim wins the race atomically and marks the id in-flight (TTL’d); release drops an uncommitted claim if the handler errors, while a committed seen mark is permanent.

This is an opt-in upgrade over the three-method seen-set, not a replacement: a store that implements only seen / remember / forget still works with the helper unchanged.

The narrow at-least-once window

Whatever the backend, there is one documented gap: the handler succeeds, then remember fails (or the worker dies) before the id is recorded. A later redelivery then reprocesses. This is bounded and acceptable under “handlers are idempotent” — a redelivery simply re-runs harmlessly. Closing it entirely requires writing the dedupe record inside the handler’s own database transaction — the transactional mode (ADR-0022), the consumer-side sibling of the transactional outbox on the produce side: the PHP PdoStore / Go postgres.Store above are the stepping stone, since a DB-backed remember can run on the same connection as the handler’s write. The seen-set store is the dependency-free default; the transactional store is the opt-in upgrade for callers who run their writes through a DB the store can join.

Cross-SDK conformance

The dedupe contract is locked by the broker-free cross-SDK conformance suite — the idempotency block in manifest.json, vendored into every core SDK. It is transport-agnostic: pure envelope + expected-outcome assertions over a single in-process store, so every SDK’s helper is proven to make the same decision.

• dedup_key — the key is meta.id verbatim; two deliveries with the same meta.id collapse to one effect, distinct ids never collapse.
• sequences — ordered delivery scenarios, each asserting whether a delivery runs the handler or is skipped, and the total side-effects after the sequence:
  • a duplicate delivery runs once;
  • an at-least-once redelivery storm is a no-op;
  • distinct ids each run;
  • a throwing handler leaves the id unmarked, so a later redelivery runs it (retry survives the guard);
  • a missing meta.id fails open (runs every time);
  • forget allows a re-run.

Because the assertions are the decision (run vs. skip) over the in-memory reference store, the block is store-agnostic — a persistent backend that honours the same three-method contract passes it unchanged.

Per-SDK entry points

The helper is optional and identical in spirit everywhere — same Store shape, same seen-set semantics, same fail-open on a missing meta.id.

For a runnable, broker-free walkthrough (no Redis, no Docker) see the in-memory idempotency example; for the cross-language Python → Go version over Redis, the idempotency-payments example.

This guard stops an accidental duplicate from re-running an effect. To stop an intended replay (off a DLQ) from re-firing effects that already ran, see DLQ redrive & replay-bypass — the complementary primitive. The producer-side sibling that makes the message commit once is the Transactional Outbox. To keep one trace across the deduped hop, see Observability (OpenTelemetry).

Back to the Idempotency contract.