Read this tutorial on Signadot docs.

Sandboxes route a request to the right version of a service by its routing key, which travels in request headers. Change data capture breaks that chain twice. The service that handles the request only writes a row, and Debezium publishes the event later from the database's write-ahead log, without ever seeing the request.

This tutorial shows how to get the routing key through anyway. An orders service writes orders to Postgres, Debezium publishes each change to Kafka, and a fulfillment consumer reads the changes and calls a notifier. You fork each service into a sandbox and watch every change reach the right version of each service. Postgres, Debezium and Kafka stay shared: you never fork them.

Full source: signadot/examples/debezium-cdc-tutorial

How the routing key gets through the database

The request reaches orders with the routing key in its baggage header. The key is stored on the row, comes out of Postgres with the change, and reaches the consumers as Kafka headers. It makes three hops:

  1. Into the database. The orders service sets the request's OpenTelemetry context (traceparent and baggage) as transaction-local Postgres settings. A trigger copies them into two columns, otel_traceparent and otel_baggage, on every insert and update.
  2. Out of the database. Debezium captures the two columns with the rest of the row, and a built-in Kafka Connect transform moves them into the Kafka headers traceparent and baggage.
  3. Out of Kafka. The baseline consumer reads through the consumer group fulfillment, and a sandbox reads through its own group, fulfillment-<routing key>. Each one asks the Signadot Routes API which keys belong to sandboxes. The baseline skips those keys, and a sandbox processes only its own.

All of this lives in platform_cdc/, a small library of the kind a platform team would ship. The app code is plain SQLAlchemy, Flask and requests.

Before you begin

Prerequisites

  • A Kubernetes cluster with the Signadot Operator installed.
  • The Signadot CLI, logged in with signadot auth login.
  • kubectl pointed at that cluster, and jq.
  • Docker, and a container registry your cluster can pull from.

1. Build the image

The three services run from one image. Build it and push it to your registry:

git clone https://github.com/signadot/examples.git cd examples/debezium-cdc-tutorial export IMAGE=docker.io/your-user/debezium-cdc-demo:latest # your registry docker build -t $IMAGE . docker push $IMAGE 

Note: Any registry your cluster can pull from works

docker.io/your-user/... is a placeholder. When you build images of your own services this way, push them to a private repository and give the cluster pull access to it, for example with an image pull secret.

If your cluster's nodes have a different CPU architecture than your machine (for example an Apple silicon Mac and an x86 cluster), add --platform linux/amd64 to docker build. On a local cluster that runs images straight from your Docker daemon, you can skip the push.

2. Deploy the demo

Point the manifests at your image, then deploy Postgres, Kafka, Debezium (Kafka Connect) and the three services into the debezium-demo namespace. The migrate Job creates the schema:

sed -i.orig "s|image: debezium-cdc-demo:latest|image: $IMAGE|" k8s/*.yaml kubectl create namespace debezium-demo kubectl apply -f k8s/ kubectl -n debezium-demo wait --for=condition=complete job/migrate --timeout=300s kubectl -n debezium-demo wait --for=condition=available deploy --all --timeout=300s 

Connect your machine to the cluster, so that cluster service names resolve locally (this asks for your password):

export CLUSTER=my-cluster # your cluster's name, from: signadot cluster list signadot local connect --cluster=$CLUSTER 

Register the Debezium connector for the orders table through the Kafka Connect REST API:

curl -X PUT http://debezium.debezium-demo.svc:8083/connectors/orders/config \ -H 'Content-Type: application/json' -d @debezium/orders-connector.json curl -s http://debezium.debezium-demo.svc:8083/connectors/orders/status | jq .tasks 

The task's state is RUNNING after a few seconds.

3. Look at the code

Three small pieces carry the routing key. The orders service creates its engine with platform_cdc.create_engine, a drop-in sqlalchemy.create_engine, and writes rows as usual (platform_cdc/db.py has the engine and the trigger):

app/orders.py

engine = platform_cdc.create_engine(DATABASE_URL) ... with engine.begin() as conn: conn.execute(text("INSERT INTO orders (id, item, status) VALUES (:id, :item, 'new')"), {"id": order["id"], "item": order["item"]}) 

The migration tracks each captured table once. That adds the two context columns and the trigger that fills them:

app/migrate.py

platform_cdc.track_table(conn, "orders") 

The connector (debezium/orders-connector.json) moves the columns into Kafka headers, the same headers a service that published the event itself would have set:

debezium/orders-connector.json

"transforms": "unwrap,context", "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState", "transforms.context.type": "org.apache.kafka.connect.transforms.HeaderFrom$Value", "transforms.context.fields": "otel_traceparent,otel_baggage", "transforms.context.headers": "traceparent,baggage", "transforms.context.operation": "move", 

The fulfillment service hands a handler to platform_cdc.ChangeConsumer, which picks the consumer group, checks each event's routing key, and makes the event's context current while the handler runs, so the call to the notifier carries the key:

app/fulfillment.py

consumer = platform_cdc.ChangeConsumer( KAFKA_BROKERS, topic=ORDERS_TOPIC, group=CONSUMER_GROUP, on_ready=lambda: pathlib.Path("/tmp/ready").touch()) consumer.run(handle) 

4. Create an order without a routing key

ORDERS=http://orders.debezium-demo.svc:4000 curl -X POST $ORDERS/api/orders -H 'Content-Type: application/json' \ -d '{"id": "o1", "item": "book"}' 

Every service writes what it did to an event log that the orders service serves. Define a helper that shows which workload handled each step of an order. The first change event takes a few seconds, while Debezium creates the topic:

trail() { curl -s $ORDERS/api/events | jq -r --arg id "$1" '.events[] | select(.context.orderID == $id) | "\(.context.baselineWorkload) (\(.context.sandboxName | if . == "" then "baseline" else . end)): \(.logEntry)"' } trail o1 

Output

orders (baseline): Inserted order into Postgres fulfillment (baseline): Consumed change event (op c) with fulfillment v1: "order o1 (book) is new" notifier (baseline): Sent notification: "order o1 (book) is new" 

5. Create the sandboxes

Each sandbox forks one service. The fulfillment sandbox also sets FULFILLMENT_VERSION=v2, the "change" under test, which uppercases item names:

signadot/sandboxes/fulfillment.yaml

name: debezium-fulfillment spec: description: "fulfillment sandbox for the Debezium CDC demo (fulfillment v2)" cluster: "@{cluster}" labels: demo: debezium-cdc forks: - forkOf: kind: Deployment namespace: debezium-demo name: fulfillment customizations: env: # The change under test: v2 uppercases item names.  - name: FULFILLMENT_VERSION value: v2 

Apply the three sandboxes and a route group that bundles them under one routing key:

signadot sandbox apply -f signadot/sandboxes/fulfillment.yaml --set cluster=$CLUSTER signadot sandbox apply -f signadot/sandboxes/notifier.yaml --set cluster=$CLUSTER signadot sandbox apply -f signadot/sandboxes/orders.yaml --set cluster=$CLUSTER signadot routegroup apply -f signadot/routegroups/debezium-demo.yaml --set cluster=$CLUSTER 

Save the routing keys:

FULFILLMENT_KEY=$(signadot sandbox get debezium-fulfillment -o json | jq -r .routingKey) NOTIFIER_KEY=$(signadot sandbox get debezium-notifier -o json | jq -r .routingKey) ORDERS_KEY=$(signadot sandbox get debezium-orders -o json | jq -r .routingKey) GROUP_KEY=$(signadot routegroup get debezium-demo -o json | jq -r .routingKey) 

6. Create orders with routing keys

Define a helper that creates an order with a routing key in the baggage header, as a preview URL or the Signadot browser extension would:

order() { curl -s -o /dev/null -X POST $ORDERS/api/orders -H 'Content-Type: application/json' \ -H "baggage: sd-routing-key=$2" -d "{\"id\": \"$1\", \"item\": \"book\"}" } 

Create one order per routing key:

order o2 $FULFILLMENT_KEY order o3 $NOTIFIER_KEY order o4 $ORDERS_KEY order o5 $GROUP_KEY sleep 2; for id in o2 o3 o4 o5; do trail $id; echo; done 

Output

orders (baseline): Inserted order into Postgres fulfillment (debezium-fulfillment): Consumed change event (op c) with fulfillment v2: "order o2 (BOOK) is new" notifier (baseline): Sent notification: "order o2 (BOOK) is new" orders (baseline): Inserted order into Postgres fulfillment (baseline): Consumed change event (op c) with fulfillment v1: "order o3 (book) is new" notifier (debezium-notifier): Sent notification: "order o3 (book) is new" orders (debezium-orders): Inserted order into Postgres fulfillment (baseline): Consumed change event (op c) with fulfillment v1: "order o4 (book) is new" notifier (baseline): Sent notification: "order o4 (book) is new" orders (debezium-orders): Inserted order into Postgres fulfillment (debezium-fulfillment): Consumed change event (op c) with fulfillment v2: "order o5 (BOOK) is new" notifier (debezium-notifier): Sent notification: "order o5 (BOOK) is new" 

Each change reached exactly the services its routing key selects:

Routing key orders fulfillment notifier
none (o1) baseline baseline (v1) baseline
fulfillment sandbox (o2) baseline sandbox (v2) baseline
notifier sandbox (o3) baseline baseline (v1) sandbox
orders sandbox (o4) sandbox baseline (v1) baseline
route group (o5) sandbox sandbox (v2) sandbox

The notifier sandbox (o3) is the clearest proof: the baseline consumer's call landed on the sandbox, so the routing key made it through Postgres, Debezium and Kafka.

The key belongs to the write, not the row

Order o2 was created through the fulfillment sandbox. Ship it with a request that has no routing key. The trigger clears the stored key, and the baseline handles this change:

curl -s -o /dev/null -X POST $ORDERS/api/orders/o2/ship; sleep 2; trail o2 | tail -3 

Output

orders (baseline): Updated order in Postgres fulfillment (baseline): Consumed change event (op u) with fulfillment v1: "order o2 (book) is shipped" notifier (baseline): Sent notification: "order o2 (book) is shipped" 

7. See the key at each hop

The rows hold the context of the request that last wrote them:

kubectl -n debezium-demo exec deploy/postgres -- \ psql -U postgres -d demo -c "SELECT id, status, otel_baggage FROM orders ORDER BY id" 

Output

 id | status | otel_baggage ----+---------+------------------------------ o1 | new | o2 | shipped | o3 | new | sd-routing-key=u5h40ssk4ykjd o4 | new | sd-routing-key=77nkpp3435chn o5 | new | sd-routing-key=mlgky8fm7lb1n 

In Kafka, the context is in the headers and no longer in the event's value:

kubectl -n debezium-demo exec deploy/kafka -- /opt/kafka/bin/kafka-console-consumer.sh \ --bootstrap-server localhost:9092 --topic demo.public.orders --from-beginning \ --timeout-ms 5000 --property print.headers=true 2>/dev/null | sed 's/__debezium[^,]*,//g' 

Output

... traceparent:00-de9f...,baggage:sd-routing-key=mlgky8fm7lb1n {"id":"o5","item":"book","status":"new","__deleted":"false","__op":"c"} traceparent:00-c9f7...,baggage:null {"id":"o2","item":"book","status":"shipped","__deleted":"false","__op":"u"} 

The baseline consumer's log shows each decision:

kubectl -n debezium-demo logs deploy/fulfillment | grep -E "processing|skipped" 

Output

... processing demo.public.orders[0]@0 (routing key None) ... skipped demo.public.orders[0]@1 (routing key 22gurfl1k6v6s) ... processing demo.public.orders[0]@2 (routing key u5h40ssk4ykjd) ... processing demo.public.orders[0]@3 (routing key 77nkpp3435chn) ... skipped demo.public.orders[0]@4 (routing key mlgky8fm7lb1n) ... processing demo.public.orders[0]@5 (routing key None) 

Your routing keys will differ.

8. Clean up

Delete the route group and the sandboxes:

signadot routegroup delete debezium-demo signadot sandbox delete debezium-fulfillment signadot sandbox delete debezium-notifier signadot sandbox delete debezium-orders 

Kafka keeps a consumer group's offsets after its consumer goes away, so the fulfillment sandbox's group is still there. With a Kafka you keep, delete it yourself:

kubectl -n debezium-demo exec deploy/kafka -- /opt/kafka/bin/kafka-consumer-groups.sh \ --bootstrap-server localhost:9092 --delete --group fulfillment-$FULFILLMENT_KEY 

Then remove the demo and disconnect:

kubectl delete namespace debezium-demo signadot local disconnect 

Things to know

  • Writes without a context go to the baseline. A write that doesn't come from a request, or comes from a client that doesn't use the platform engine (a psql session, a batch job), clears the two columns, so its change event has no routing key.
  • Deletes go to the baseline. A deleted row has nowhere to store the context of the request that deleted it. If sandboxes need to see deletes, use a soft delete (an update), or write the event to an outbox table that is tracked the same way.
  • Sandboxes share the database. A sandboxed orders service writes to the baseline's table. The stored routing key tells you which rows came from which sandbox. To give a sandbox its own database instead, see Using sandboxes with databases.

Frequently asked questions

How do you test a Debezium CDC pipeline without a separate database and Kafka per environment?

Store the routing key with each row, let Debezium move it into Kafka headers, and have consumers filter on it. A Postgres trigger stamps the OpenTelemetry baggage of the writing request on the row, a Kafka Connect transform turns those columns into the baggage and traceparent headers, and each consumer checks the key against the Signadot Routes API. Sandboxes then share one Postgres, one Debezium connector and one Kafka topic with the baseline.

Why does change data capture need more than header propagation?

Because the service that handles the request never publishes the event. It only writes a row. Debezium publishes the event later from the write-ahead log, and it has never seen the request or its headers. The routing key has to be written into the database with the row to reach the change event.

Does the application code have to change?

Very little. The service creates its SQLAlchemy engine with platform_cdc.create_engine, a migration calls platform_cdc.track_table once per captured table, and consumers pass a handler to platform_cdc.ChangeConsumer. The SQL the app runs never mentions the routing key, and the Debezium connector needs six lines of transform configuration.

What happens to deletes and to writes made outside a request?

Both go to the baseline. A deleted row has nowhere to store the context of the request that deleted it, and a write from psql or a batch job has no context, so the trigger clears the columns. If sandboxes need to see deletes, use a soft delete or an outbox table that is tracked the same way.

Does this work with the outbox pattern?

Yes. Track the outbox table with platform_cdc.track_table instead of the business table. Each outbox row then carries the routing key of the request that wrote it, and Debezium moves it into the Kafka headers in the same way.

See also