Skip to content

Commit 569b3d3

Browse files
Akeboshiwindclaude
authored and
refset
committed
feat(debezium): add full e2e demo using docker-compose
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 87d1172 commit 569b3d3

5 files changed

Lines changed: 295 additions & 8 deletions

File tree

‎debezium/README.md‎

Lines changed: 23 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,20 @@ mise run demo # Installs MariaDB, starts it, runs CDC
3333

3434
The module also includes a Debezium Server sink connector for deployment with standalone Debezium Server.
3535

36+
### [cdc-demo](./cdc-demo/)
37+
38+
**Docker Compose end-to-end demo** - Full stack with Postgres, Debezium Connect, Kafka, and XTDB's built-in `debezium-cdc` module.
39+
40+
- Postgres source (WAL-based CDC)
41+
- Kafka as the message bus
42+
- Uses XTDB's native CDC ingestion (no custom code)
43+
- Fully self-contained via docker-compose
44+
45+
```bash
46+
cd cdc-demo
47+
docker compose up -d
48+
```
49+
3650
## Key Concepts
3751

3852
### Schema-less Ingestion
@@ -63,14 +77,15 @@ The demos handle both full Debezium envelope format and the flattened format (vi
6377

6478
## Comparison
6579

66-
| Feature | Static JSON | Live CDC (debezium-xtdb) |
67-
|---------|-------------|--------------------------|
68-
| Real database | No | Yes (MySQL/MariaDB) |
69-
| Kafka required | No | No |
70-
| CDC engine | None | Debezium Embedded |
71-
| Latency | N/A | Sub-second |
72-
| Setup complexity | Minimal | Medium (MariaDB install) |
73-
| Best for | Learning | Development/Testing |
80+
| Feature | Static JSON | Live CDC (debezium-xtdb) | Docker Compose (cdc-demo) |
81+
|---------|-------------|--------------------------|---------------------------|
82+
| Real database | No | Yes (MySQL/MariaDB) | Yes (Postgres) |
83+
| Kafka required | No | No | Yes |
84+
| CDC engine | None | Debezium Embedded | Debezium Connect |
85+
| XTDB ingestion | Custom (JDBC) | Custom (JDBC) | Built-in `debezium-cdc` |
86+
| Latency | N/A | Sub-second | Sub-second |
87+
| Setup complexity | Minimal | Medium (MariaDB install) | Medium (Docker) |
88+
| Best for | Learning | Development/Testing | Production-like demo |
7489

7590
## Architecture
7691

‎debezium/cdc-demo/README.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# Debezium CDC Demo
2+
3+
Demonstrates end-to-end Change Data Capture from Postgres into XTDB via Debezium and Kafka.
4+
5+
## Prerequisites
6+
7+
Build the Docker image (includes the debezium module):
8+
9+
```bash
10+
./docker/scripts/build-aws-image.sh --clean
11+
```
12+
13+
## Start the stack
14+
15+
```bash
16+
cd debezium/cdc-demo
17+
docker compose up -d
18+
```
19+
20+
Wait ~30s for all services to stabilise.
21+
Check status with `docker compose ps` — `debezium-init` and `minio-setup` will show as exited (expected, they're one-shot).
22+
23+
## Run the demo
24+
25+
To submit transactions to Postgres:
26+
27+
```bash
28+
PGPASSWORD=postgres psql -h localhost -p 5434 -U postgres -d sourcedb
29+
```
30+
31+
To watch messages on the kafka topic:
32+
33+
```bash
34+
docker compose exec kafka kafka-console-consumer --bootstrap-server kafka:29092 --topic sourcedb.public.cdc_demo --from-beginning 2>/dev/null | jq .
35+
```
36+
37+
To watch for changes in XTDB:
38+
```bash
39+
watch -n 0.5 "psql -h localhost -p 5433 -U xtdb -d xtdb -c \"SELECT *, _valid_time FROM public.cdc_demo FOR ALL VALID_TIME ORDER BY _id, _valid_from\""
40+
```
41+
42+
### 1. Create a table and insert data in Postgres
43+
44+
```sql
45+
CREATE TABLE cdc_demo (_id INT PRIMARY KEY, name TEXT, email TEXT);
46+
INSERT INTO cdc_demo VALUES (1, 'Alice', 'alice@example.com');
47+
INSERT INTO cdc_demo VALUES (2, 'Bob', 'bob@example.com');
48+
```
49+
50+
### 2. More mutations
51+
52+
```sql
53+
UPDATE cdc_demo SET email = 'alice-new@example.com' WHERE _id = 1;
54+
DELETE FROM cdc_demo WHERE _id = 2;
55+
INSERT INTO cdc_demo VALUES (3, 'Charlie', 'charlie@example.com');
56+
```
57+
58+
## Tear down
59+
60+
```bash
61+
docker compose down -v
62+
```

‎debezium/cdc-demo/dc_config.yaml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
server:
2+
host: '*'
3+
port: 5432
4+
5+
flightSql:
6+
host: '*'
7+
port: 9832
8+
9+
logClusters:
10+
kafkaCluster: !Kafka
11+
bootstrapServers: !Env KAFKA_BOOTSTRAP_SERVERS
12+
13+
log: !Kafka
14+
cluster: kafkaCluster
15+
topic: !Env XTDB_LOG_TOPIC
16+
17+
storage: !Remote
18+
objectStore: !S3
19+
bucket: !Env XTDB_S3_BUCKET
20+
prefix: "xtdb-object-store"
21+
endpoint: !Env XTDB_S3_ENDPOINT
22+
credentials:
23+
accessKey: !Env ACCESS_KEY
24+
secretKey: !Env SECRET_KEY
25+
pathStyleAccessEnabled: true
26+
27+
diskCache:
28+
path: /var/lib/xtdb/buffers
29+
30+
healthz:
31+
host: '*'
32+
port: 8080

‎debezium/cdc-demo/docker-compose.yaml‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
version: '3'
2+
services:
3+
minio:
4+
image: minio/minio
5+
ports:
6+
- "9000:9000"
7+
- "8090:8090"
8+
environment:
9+
- MINIO_ROOT_USER=minioadmin
10+
- MINIO_ROOT_PASSWORD=minioadmin
11+
volumes:
12+
- minio_data:/data
13+
command: server /data --console-address ":8090"
14+
healthcheck:
15+
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
16+
interval: 5s
17+
timeout: 5s
18+
retries: 3
19+
20+
# a service simply to create the bucket
21+
minio-setup:
22+
image: minio/mc
23+
depends_on:
24+
minio:
25+
condition: service_healthy
26+
entrypoint: >
27+
/bin/sh -c "
28+
mc alias set myminio http://minio:9000 minioadmin minioadmin &&
29+
mc mb --ignore-existing myminio/xtdb-bucket &&
30+
echo 'MinIO bucket created successfully!'"
31+
32+
kafka:
33+
image: confluentinc/cp-kafka:7.7.1
34+
expose:
35+
- 9092
36+
- 9999
37+
- 29092
38+
- 29093
39+
ports:
40+
- 29092:29092
41+
- 29093:29093
42+
environment:
43+
CLUSTER_ID: "1"
44+
KAFKA_PROCESS_ROLES: "broker,controller"
45+
KAFKA_NODE_ID: "1"
46+
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT"
47+
KAFKA_ADVERTISED_LISTENERS: "PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092"
48+
KAFKA_CONTROLLER_QUORUM_VOTERS: "1@kafka:29093"
49+
KAFKA_LISTENERS: "PLAINTEXT://kafka:29092,CONTROLLER://kafka:29093,PLAINTEXT_HOST://0.0.0.0:9092"
50+
KAFKA_CONTROLLER_LISTENER_NAMES: "CONTROLLER"
51+
KAFKA_INTER_BROKER_LISTENER_NAME: "PLAINTEXT"
52+
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: "1"
53+
54+
# source postgres
55+
postgres:
56+
image: postgres:17-alpine
57+
ports:
58+
- "5434:5432"
59+
environment:
60+
POSTGRES_USER: postgres
61+
POSTGRES_PASSWORD: postgres
62+
POSTGRES_DB: sourcedb
63+
command: ["postgres", "-c", "wal_level=logical"]
64+
healthcheck:
65+
test: ["CMD-SHELL", "pg_isready -U postgres"]
66+
interval: 5s
67+
timeout: 5s
68+
retries: 5
69+
70+
# captures PG WAL → Kafka
71+
debezium-connect:
72+
image: quay.io/debezium/connect:3.0
73+
depends_on:
74+
kafka:
75+
condition: service_started
76+
postgres:
77+
condition: service_healthy
78+
ports:
79+
- "8083:8083"
80+
environment:
81+
BOOTSTRAP_SERVERS: kafka:29092
82+
GROUP_ID: "1"
83+
CONFIG_STORAGE_TOPIC: debezium_configs
84+
OFFSET_STORAGE_TOPIC: debezium_offsets
85+
STATUS_STORAGE_TOPIC: debezium_statuses
86+
87+
debezium-init:
88+
image: curlimages/curl
89+
depends_on:
90+
- debezium-connect
91+
- postgres
92+
volumes:
93+
- ./register-connector.sh:/register-connector.sh:ro
94+
entrypoint: ["/bin/sh", "/register-connector.sh"]
95+
96+
# main XTDB node (queryable via pgwire on :5433)
97+
xtdb:
98+
image: xtdb/xtdb-aws:latest
99+
depends_on:
100+
- kafka
101+
- minio-setup
102+
expose:
103+
- 8080
104+
- 5432
105+
- 9832
106+
ports:
107+
- "8081:8080"
108+
- "5433:5432"
109+
- "9832:9832"
110+
environment:
111+
AWS_REGION: "aws-iso-global"
112+
AWS_S3_FORCE_PATH_STYLE: "true"
113+
AWS_S3_USE_VIRTUAL_HOSTING: "false"
114+
XTDB_NODE_ID: "xt-node-1"
115+
KAFKA_BOOTSTRAP_SERVERS: "kafka:29092"
116+
XTDB_LOG_TOPIC: "xt_log"
117+
XTDB_S3_BUCKET: "xtdb-bucket"
118+
XTDB_S3_ENDPOINT: "http://minio:9000"
119+
ACCESS_KEY: "minioadmin"
120+
SECRET_KEY: "minioadmin"
121+
command: ["-f", "/config/dc_config.yaml"]
122+
volumes:
123+
- ./dc_config.yaml:/config/dc_config.yaml
124+
125+
# XTDB CDC node
126+
xtdb-cdc:
127+
image: xtdb/xtdb-aws:latest
128+
depends_on:
129+
- kafka
130+
- minio-setup
131+
- debezium-init
132+
environment:
133+
AWS_REGION: "aws-iso-global"
134+
AWS_S3_FORCE_PATH_STYLE: "true"
135+
AWS_S3_USE_VIRTUAL_HOSTING: "false"
136+
XTDB_NODE_ID: "xt-cdc-node"
137+
KAFKA_BOOTSTRAP_SERVERS: "kafka:29092"
138+
XTDB_LOG_TOPIC: "xt_log"
139+
XTDB_S3_BUCKET: "xtdb-bucket"
140+
XTDB_S3_ENDPOINT: "http://minio:9000"
141+
ACCESS_KEY: "minioadmin"
142+
SECRET_KEY: "minioadmin"
143+
command: ["debezium-cdc", "--kafka-cluster", "kafkaCluster", "-t", "sourcedb.public.cdc_demo", "-f", "/config/dc_config.yaml"]
144+
volumes:
145+
- ./dc_config.yaml:/config/dc_config.yaml
146+
147+
volumes:
148+
minio_data:

‎debezium/cdc-demo/register-connector.sh‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
#!/bin/sh
2+
set -e
3+
4+
CONNECT_URL="http://debezium-connect:8083"
5+
6+
echo "Waiting for Debezium Connect to be ready..."
7+
until curl -sf "$CONNECT_URL/connectors" > /dev/null 2>&1; do
8+
sleep 2
9+
done
10+
echo "Debezium Connect is ready."
11+
12+
curl -sf -X POST "$CONNECT_URL/connectors" \
13+
-H "Content-Type: application/json" \
14+
-d '{
15+
"name": "sourcedb-connector",
16+
"config": {
17+
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
18+
"database.hostname": "postgres",
19+
"database.port": "5432",
20+
"database.user": "postgres",
21+
"database.password": "postgres",
22+
"database.dbname": "sourcedb",
23+
"topic.prefix": "sourcedb",
24+
"schema.include.list": "public",
25+
"plugin.name": "pgoutput"
26+
}
27+
}'
28+
29+
echo ""
30+
echo "Connector registered successfully."

0 commit comments

Comments
 (0)