Transactional Outbox

Idempotency deduplicates on the consumer side. The outbox is its mirror on the producer side: it removes the dual write — the crash window between committing your business row and publishing the message — so the message commits atomically with the business data, then a separate relay hands it to the broker (ADR-0029).

Status: Authoritative · Helper layer · envelope frozen at schema_version: 1

Nothing here touches the wire. The outbox stores the existing encoded envelope verbatim and the relay publishes those exact bytes — trace_id preserved, byte-compatible. The outbox’s own columns (id, status, attempts) are bookkeeping around the envelope, never on it. The envelope stays frozen at schema_version: 1.

The dual write, and why a transaction can’t span it

A plain producer does two things that must both happen or neither: commit the business row (INSERT INTO orders …) and publish the message to the broker. They are two independent systems. A crash between them leaves the order saved with no message, or a message sent for an order that rolled back. There is no distributed transaction to lean on — BabelQueue is broker-agnostic and deliberately avoids one (GR-7), and no broker offers a commit the database can see.

The outbox pattern removes the dual write by making the publish a local write first:

1. In the same database transaction as the business row, persist the encoded envelope into an outbox table. Commit both together — they succeed or roll back atomically.
2. A separate relay reads the durable outbox rows afterwards, publishes each to the broker, and marks it published. The handoff to the broker is now a local problem (read a row, publish, mark it), not a cross-system one.

The honest scope is exactly-once handoff into the broker, then at-least-once on the wire as always: a crash between publish and mark-published re-publishes the row on the next pass. That redelivery is deduped by the consumer on meta.id — idempotency is the consumer-side mirror that closes the loop.

The OutboxStore

Every SDK exposes the same four-method persistence contract. The core defines it and binds to no DB driver (GR-7); a concrete adapter over your database is the caller’s:

The transaction boundary is the caller’s. save() runs on the transaction you already opened around your business write; you commit both together. The store never begins or commits anything — that is the whole point, and the reason no DB driver enters the core.

The Outbox writer

The writer is tiny: Outbox.write(envelope) encodes the envelope via the frozen EnvelopeCodec (bytes unchanged), captures meta.queue (else "default"), and delegates to OutboxStore.save(). It does not begin or commit anything. You call it inside your own transaction, right beside your business write:

begin transaction
    INSERT INTO orders (…)            -- business row
    outbox.write(envelope)           -- the message, same transaction
commit                                -- both, or neither

The OutboxRelay

The relay drives the broker handoff through the existing publish-only Transport seam:

• flush() publishes one batch of pending rows. Each row is marked published only after the transport accepts it; a failing publish is caught (markFailed, row left pending) with a bounded linear backoff, and the batch continues — one poison row never blocks the others. The sleeper is injectable, so tests run instantly.
• drain() loops flush while a pass makes progress, with a safety ceiling.

The relay publishes the stored bytes verbatim — it never decodes, rebuilds, or re-encodes the envelope. So trace_id is preserved end-to-end (GR-4) and the published bytes are byte-identical to what the producer encoded (GR-1/GR-5).

The in-memory reference store

Every core ships an InMemoryOutboxStore for tests and single-process demos. It honours the four-method contract over a process-local, insertion-ordered map (oldest-first) but has no real transaction and does not claim/lock rows. Production needs a DB-backed adapter — the InitORM outbox example is the reference for one (the outbox-table DDL, a produce.php that does the business write and outbox->write in one $db->transaction, and a relay.php), keeping the core itself DB-free.

Go

The helper is the …/outbox subpackage of the core module (stdlib only, like idempotency/schema). outbox.New(store) is the writer; outbox.NewRelay(transport, store, opts) is the relay:

import (
	babelqueue "github.com/babelqueue/babelqueue-go"
	"github.com/babelqueue/babelqueue-go/outbox"
)

store := outbox.NewInMemoryStore() // tests / single process
box := outbox.New(store)

// inside the caller's own DB transaction, beside the business write:
env, _ := babelqueue.Make("urn:babel:orders:created",
	map[string]any{"order_id": 1042}, babelqueue.WithQueue("orders"))
id, err := box.Write(env) // encodes via the frozen codec, calls Store.Save — no commit

// later, a relay drains the durable rows to the broker:
relay := outbox.NewRelay(transport, store, outbox.Options{})
res, err := relay.Drain(ctx, 0) // res.Published / res.Failed

Store is the interface (Save / FetchUnpublished / MarkPublished / MarkFailed); a production fleet implements it over Postgres/MySQL — binding Save to the caller’s open transaction — and passes it to New and NewRelay unchanged.

PHP

BabelQueue\Outbox\Outbox::write() mirrors the Go writer; OutboxRelay drains through the publish-only Transport:

use BabelQueue\Outbox\Outbox;
use BabelQueue\Outbox\OutboxRelay;
use BabelQueue\Outbox\InMemoryOutboxStore;

$store = new InMemoryOutboxStore(); // tests / single process
$outbox = new Outbox($store);

// inside your own $db->transaction(...), beside the business write:
$id = $outbox->write($envelope); // encodes via EnvelopeCodec, calls $store->save()

// later: a relay hands the durable rows to the broker
$result = (new OutboxRelay($transport, $store))->drain();
// $result->published / $result->failed

A production store implements BabelQueue\Outbox\OutboxStore (save / fetchUnpublished / markPublished / markFailed) over a PDO/ORM connection, with save() running on the caller’s open transaction — see the InitORM example’s InitOrmOutboxStore.

Per-SDK

The outbox is optional and identical in spirit everywhere — same four-method OutboxStore, same caller-owned transaction, same write / flush / drain, same verbatim-bytes relay. It shipped across all six SDK cores (per-SDK MINORs; the envelope stayed frozen).

Two idiomatic differences to expect: .NET is async throughout (WriteAsync / FetchUnpublishedAsync / FlushAsync / DrainAsync, CancellationToken-threaded) and its store interface is IOutboxStore; Python’s method names are snake-case (fetch_unpublished / mark_published / mark_failed) and its relay backoff is in seconds where the others are in milliseconds. The broker seam the relay forwards through is the SDK’s existing publish surface — a Transport (Go/PHP/Python), an OutboxTransport (Node/Java), or an OutboxPublisher delegate (.NET).

The DB-backed adapter stays the caller’s in every SDK — the cores ship only the in-memory reference, exactly as on the PHP side. Relay concurrency (claim/lock so two relays don’t double-publish), and retention/archival of published rows, are the adapter’s concern.

See each SDK’s reference for wiring a store: Go, PHP, Python, Node, Java, .NET.

The outbox guarantees the message is sent once you commit; idempotency guarantees its effect fires once on the consumer despite the at-least-once redelivery the relay’s re-publish can cause. Use them together. To keep one trace across the produce→consume hop, see Observability (OpenTelemetry).