Per-URN Schema Validation

The wire contract freezes the envelope, but deliberately keeps data as “pure JSON the caller validates” — the business payload differs per URN, so the contract cannot fix its shape. URN naming §6 recommends that teams keep a checked-in, per-URN registry with a JSON Schema for each message’s data. This page is the cross-SDK contract for enforcing those schemas (ADR-0024) and the babelqueue-registry / bqschema tooling that governs them.

Status: Authoritative · Opt-in helper + tooling · envelope frozen at schema_version: 1

This is a tooling layer over data, not a wire change. The envelope is untouched; what is validated is the data block, against a schema you register for a URN. It is opt-in: a URN with no registered schema is never validated.

The model: a Provider per URN

Every SDK exposes a Provider (interface / protocol) that answers one question — what is the JSON Schema for this URN’s data? — returning the schema, or “none registered” so the caller skips validation. Two reference providers ship in each SDK:

• MapProvider — in-memory, for tests and for embedding schemas in code.
• DirProvider — the registry bridge: it reads a babelqueue-registry registry.json manifest (a list of {urn, schema} entries) and loads each URN’s schema file lazily. This is what makes the registry’s git-governed schemas enforceable at runtime.

The validator is an intentionally small subset of JSON Schema draft-07 — type, required, properties, additionalProperties, items, enum, const, minLength, minimum — enough for real data shapes. Unknown keywords are ignored, and the verdicts match across Go, PHP, Python and bqschema, so a schema validated in CI validates identically at runtime. Zero dependencies (stdlib only).

Producer-side Check is preferred

Validation can run on either side, but the sides are not equivalent:

• Producer-side (Check / assert / validate) — preferred. Call it before publishing so invalid data never enters the queue. A violation is caught at the source, where the caller can fix it, and the bad message is never written.
• Consumer-side (Wrap) — a safety net. Wrapping a handler validates each message before it runs. But invalid data will not become valid on retry, so a poison message exhausts its attempts and is dead-lettered. That is correct fail-safe behaviour, but it spends retries on a message that can never succeed — hence producer-side Check is the recommended primary gate, with Wrap as defence in depth.

Go

schema.Check(provider, urn, data) is the producer guard; schema.Wrap(provider, handler) is the consumer safety net. schema.Validate(provider, env) is the envelope form of Check. The package is in the core module (stdlib only):

import "github.com/babelqueue/babelqueue-go/schema"

// DirProvider bridges a babelqueue-registry manifest to runtime validation
provider, err := schema.NewDirProvider("registry.json")
if err != nil {
	return err
}

// producer-side (preferred): keep invalid data out of the queue
if err := schema.Check(provider, "urn:babel:orders:created",
	map[string]any{"order_id": 1042, "amount": 99.90}); err != nil {
	return err // errors.Is(err, schema.ErrInvalidPayload)
}

// consumer-side safety net
app.Handle("urn:babel:orders:created", schema.Wrap(provider, handler))

schema.NewMapProvider(map[string][]byte{...}) builds an in-memory provider for tests.

PHP

BabelQueue\Schema\SchemaValidated mirrors the Go helper — assert (throwing producer guard), check (non-throwing, for branching), and wrap (consumer safety net):

use BabelQueue\Schema\DirProvider;
use BabelQueue\Schema\SchemaValidated;

$provider = new DirProvider('registry.json'); // registry bridge

// producer-side (preferred)
SchemaValidated::assert($provider, 'urn:babel:orders:created', [
    'order_id' => 1042,
    'amount'   => 99.90,
]); // throws InvalidPayloadException on mismatch

// consumer-side safety net
$dispatch->on('urn:babel:orders:created', SchemaValidated::wrap($provider, $handler));

new BabelQueue\Schema\MapProvider([...]) is the in-memory provider.

Python

babelqueue.schema mirrors the same surface — validate (raising producer guard), check (non-raising), and wrap (consumer safety net, taking the URN explicitly since a Python handler receives data positionally):

from babelqueue.schema import DirProvider, validate, wrap

provider = DirProvider("registry.json")  # registry bridge

# producer-side (preferred)
validate(provider, "urn:babel:orders:created",
         {"order_id": 1042, "amount": 99.90})  # raises InvalidPayloadError on mismatch

# consumer-side safety net
app.register("urn:babel:orders:created", wrap(provider, "urn:babel:orders:created", handler))

MapProvider.from_json({...}) builds the in-memory provider from raw schema strings.

Governing schemas: babelqueue-registry and bqschema

The runtime helpers enforce schemas; the babelqueue-registry project governs them. bqschema is a CLI you run in CI — no Kafka, no service, no database. Schemas live in your git repo as files, referenced from a registry.json manifest:

{
  "schemas": [
    { "urn": "urn:babel:orders:created", "schema": "schemas/orders-created.json", "owner": "orders" }
  ]
}

go install github.com/babelqueue/babelqueue-registry/cmd/bqschema@latest

# does this message's data match the schema registered for its URN?
bqschema validate --registry registry.json messages/order-created.json

# is this schema change backward-compatible, or does it break consumers?
bqschema compat schemas/orders-created.json schemas/orders-created.v2.json

# sanity-check the registry itself (every schema parses)
bqschema check --registry registry.json

# generate an AsyncAPI 3.0 event catalog from the registry
bqschema export-asyncapi --registry registry.json -o asyncapi.json

# inventory / audit / mask the GDPR-sensitive fields declared in the registry
bqschema gdpr --registry registry.json                 # inventory the x-gdpr-sensitive paths
bqschema gdpr --registry registry.json --require       # CI gate: fail on un-annotated PII
bqschema gdpr --registry registry.json --mask msg.json # redact PII for safe logging

compat enforces the versioning policy at the data level, the same way the envelope rules work for the wire: adding an optional field is compatible; removing / renaming / retyping a field, or making an optional field required, is breaking — the signal to mint a new URN (…:created.v2) rather than mutate the existing one, so consumers can upgrade before producers. Unlike a broker-coupled schema registry, the schemas are plain git files, so there is no cold-start circular dependency and the gate works identically across Redis, RabbitMQ, SQS, Kafka, Pulsar and the rest.

GDPR-sensitive fields: declare and audit

The same registry is where PII is declared. An x-gdpr-sensitive keyword on a property (value true or a category string like "email") marks a data field as personal data. It is validation-neutral — parsed but ignored by validate and compat, so annotating an existing schema is never a breaking change — and it flows through into the AsyncAPI catalog automatically. bqschema gdpr inventories the marked paths, --require is a CI gate that fails when a PII-named field is left un-annotated, and --mask redacts a message for safe logging (one-way, registry-side — not a crypto primitive). The matching runtime encryption that reads the same keyword to protect those fields on the wire is the SDK’s job — see GDPR field encryption (ADR-0030).

Per-SDK

Validation is optional and opt-in everywhere — same Provider model, same draft-07 subset, same producer-preferred / consumer-safety-net split. The registry bridge (DirProvider) ships in Go, PHP and Python; the in-memory MapProvider is the path in Node, Java and .NET.

See each SDK’s reference and the babelqueue-registry README for bqschema details: Go, PHP, Python, Node, Java.

Continue to GDPR field encryption.