ticdc: document Debezium source.start_ts option - #23614
Conversation
Document the optional Debezium JSON source.start_ts field (v8.5.9): URI debezium-include-start-ts, TOML include-start-ts, URI precedence, DML-only scope, and uint64 PD TSO precision.
ekexium
added
type/enhancement
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
ti-chi-bot
Bot
added
missing-translation-status
📝 WalkthroughWalkthroughThe documentation adds optional inclusion of the source transaction’s original PD TSO in ChangesDebezium start timestamp
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: 🔵 Low · up to The PR is generally mergeable, but owner follow-up is needed to clarify the supported numeric range for Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Full details: Docstring CoverageExplanation No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (4 skipped: 4 unsupported.) ✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR updates TiCDC Kafka/Debezium documentation to describe the new optional source.start_ts field (introduced in v8.5.9) and how to enable it via sink URI, changefeed config, and the OpenAPI v2 config model.
Changes:
- Adds the
debezium-include-start-tssink URI parameter to the Kafka sink reference. - Documents
[sink.debezium] include-start-tsin the changefeed configuration reference andinclude_start_tsin the OpenAPI v2sink.debeziumconfig. - Expands the Debezium protocol doc with a new “Include the transaction start TSO” section and adds
payload.source.start_tsto the field table.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| ticdc/ticdc-sink-to-kafka.md | Adds a new sink URI parameter to enable emitting source.start_ts in Debezium JSON DML messages. |
| ticdc/ticdc-open-api-v2.md | Documents the new sink.debezium.include_start_ts OpenAPI v2 config field. |
| ticdc/ticdc-debezium.md | Adds a new section describing behavior/limitations and documents payload.source.start_ts in the message field table. |
| ticdc/ticdc-changefeed-config.md | Documents the new [sink.debezium] include-start-ts changefeed configuration item and its precedence rules. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| | Parameter name | Description | | ||
| |:-------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ||
| | `output_old_value` | `BOOLEAN` type. It controls whether to output the value before the row data changes. The default value is true. When it is disabled, the UPDATE event does not output the "before" field. | | ||
| | `include_start_ts` | `BOOLEAN` type. New in v8.5.9. It controls whether Debezium JSON DML messages include `source.start_ts` (the original PD TSO of the source transaction). The default value is `false`. | |
|
|
||
| - DML value messages add integer `source.start_ts` next to `commit_ts`, and the JSON schema declares the field as `int64`. | ||
| - DDL events, WATERMARK/checkpoint events, key messages, and Debezium Avro are unchanged. Setting this option with the Debezium Avro protocol is rejected. | ||
| - To roll back, disable the option. Messages produced while it is off stay byte-compatible with the previous format. |
|
|
||
| > **Note:** | ||
| > | ||
| > `start_ts` is the original uint64 PD TSO, not a millisecond timestamp. Consumers must treat it as a 64-bit integer or a decimal string. Do not parse it as a JavaScript `Number` or IEEE-754 `float64`, which cannot represent an 18-digit TSO exactly. |
| | `payload.before` | JSON | The data value before the change event of a statement. For `"c"` events, the value of the `before` field is `null`. | | ||
| | `payload.after` | JSON | The data value after the change event of a statement. For `"d"` events, the value of the `after` field is `null`. | | ||
| | `payload.source.commit_ts` | Number | The `CommitTs` value of the event. | | ||
| | `payload.source.start_ts` | Number | The start TSO of the source transaction. Present only when `debezium-include-start-ts` or `[sink.debezium] include-start-ts` is enabled. Original uint64 PD TSO, not a millisecond timestamp. | |
ekexium
added
the
translation/done
ti-chi-bot
Bot
removed
the
missing-translation-status
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 72067aab-fc9f-4682-ae05-31418d6562df
📒 Files selected for processing (4)
ticdc/ticdc-changefeed-config.mdticdc/ticdc-debezium.mdticdc/ticdc-open-api-v2.mdticdc/ticdc-sink-to-kafka.md
Included review availability: Your plan provides up to 4 included reviews per hour; 3 remain after this review.
|
|
||
| When this option is enabled: | ||
|
|
||
| - DML value messages add integer `source.start_ts` next to `commit_ts`, and the JSON schema declares the field as `int64`. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
#!/bin/bash
# Expected: encoder, schema, and API definitions agree on signed/unsigned and JSON representation.
rg -n -C 8 'include[-_]start[-_]ts|includeStartTs|start_ts|startTs|uint64|int64' . \
--glob '*.go' --glob '*.proto' --glob '*.json' --glob '*.yaml' --glob '*.toml'Repository: pingcap/docs
Length of output: 150
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- repository convention files ---'
find /tmp/coderabbit-repo-knowledge/pingcap-docs-9c73f70c -maxdepth 2 -type f -print | sort
printf '%s\n' '--- reviewed document ---'
sed -n '30,75p' ticdc/ticdc-debezium.md
printf '%s\n' '--- relevant tracked files ---'
git ls-files | rg '(^|/)(ticdc|resources|scripts)/|debezium|cdc|protocol' | head -200
printf '%s\n' '--- documentation references ---'
rg -n -C 4 'start_ts|startTs|include[-_]start[-_]ts|includeStartTs|uint64|int64|Number|Debezium Avro|protocol' ticdc resources README.md 2>/dev/null | head -300Repository: pingcap/docs
Length of output: 36589
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- repository-wide convention ---'
cat /tmp/coderabbit-repo-knowledge/pingcap-docs-9c73f70c/conventions/repo-wide.md
printf '%s\n' '--- exact document context ---'
sed -n '1,70p' ticdc/ticdc-debezium.md
printf '%s\n' '--- local option references ---'
rg -n -C 8 'include-start-ts|debezium-include-start-ts|source\.start_ts' ticdcRepository: pingcap/docs
Length of output: 22621
🌐 Web query:
pingcap tiflow include-start-ts debezium source.start_ts uint64 int64 implementation
💡 Result:
In TiCDC, the start-ts of a changefeed is a Timestamp Oracle (TSO) value [1][2]. When using the Debezium protocol for TiCDC, the protocol output includes metadata fields in the payload, such as payload.source.commit_ts [3]. Regarding your query about include-start-ts, source.start_ts, and type mappings: 1. Start-ts: This is a uint64 value representing the starting TSO in the upstream TiDB cluster [1][4]. When creating or resuming a changefeed, this value is used to specify the point from which TiCDC begins pulling data [5][6]. If not specified (or set to 0), TiCDC fetches a current TSO to start the task [1][2]. 2. Debezium protocol output: TiCDC's Debezium protocol output is designed to be compatible with downstream systems that expect the standard Debezium format, while adding specific fields to ensure data integrity in TiDB [3]. Specifically, TiCDC adds CommitTs and ClusterID to the payload's source section [3]. The field payload.source.commit_ts is included as a Number type [3]. 3. Data types: TiDB timestamps (TSOs) are 64-bit unsigned integers (uint64) [1][7]. When these values are serialized into JSON for the Debezium protocol, they are typically represented as numbers [3]. Downstream systems or custom consumers reading these JSON messages should handle these as 64-bit integers to prevent precision loss or overflow, as standard JSON numbers are often parsed as 64-bit floating-point numbers in some languages [3]. There is no standard configuration parameter named include-start-ts or source.start_ts in the TiCDC Debezium sink configuration [5][6]. The core parameter for defining the start point is the --start-ts flag used during changefeed creation [5][6]. If you are encountering references to source.start_ts in your integration, it is likely a custom mapping or a reference to the CommitTs field which serves the purpose of identifying the transaction order [3].
Citations:
- 1: https://docs.pingcap.com/tidb/stable/ticdc-faq/
- 2: https://docs.pingcap.com/tidb/v8.1/ticdc-faq/
- 3: https://docs.pingcap.com/tidb/stable/ticdc-debezium/
- 4: https://docs.pingcap.com/tidb/dev/ticdc-faq/
- 5: https://docs.pingcap.com/tidb/dev/ticdc-changefeed-config/
- 6: https://github.com/pingcap/docs/blob/master/ticdc/ticdc-changefeed-config.md
- 7: https://pkg.go.dev/github.com/pingcap/tidb/types
🏁 Script executed:
#!/bin/bash
set -eu
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT
curl -fsSL 'https://api.github.com/repos/pingcap/tiflow/git/trees/master?recursive=1' -o "$tmp"
printf '%s\n' '--- matching upstream paths ---'
jq -r '.tree[] | select(.path | test("debezium|codec|sink")) | .path' "$tmp" | rg -i 'debezium|codec' | head -200
printf '%s\n' '--- source paths containing likely option names ---'
jq -r '.tree[] | select(.type == "blob") | .path' "$tmp" | rg -i '\.(go|proto|json|yaml|toml)$' | head -300Repository: pingcap/docs
Length of output: 19063
🌐 Web query:
site:github.com/pingcap/tiflow "debezium-include-start-ts"
💡 Result:
There is no configuration parameter named debezium-include-start-ts or include.start.ts in the standard Debezium connector configuration. Based on the technical context of tools like TiCDC (a Change Data Capture tool that supports the Debezium output format), the term "start-ts" refers to a transaction start timestamp, which is a core concept in TiDB-based replication systems (such as ignore-txn-start-ts in TiCDC configuration files) [1]. If you are encountering a reference to start-ts in relation to Debezium, it is likely appearing in one of the following contexts: 1. TiCDC Configuration: When configuring a changefeed to output data in the Debezium protocol, users manage transaction timestamps using settings like ignore-txn-start-ts within the changefeed's TOML configuration file [1]. 2. Troubleshooting and Logs: System logs for data replication often include the start-ts of transactions when operations fail or to track progress, but this is not a connector-level "include" flag [2]. If you are looking for ways to include timestamps in Debezium events, Debezium naturally includes source-side timestamps (such as source.ts_ms or ts_ms) in its standard JSON or Avro message formats, which do not require a specific "include" configuration parameter.
Citations:
- 1: https://github.com/pingcap/tiflow/blob/7c3c2336f98153326912f3cf6ea2fbb7bcc4a20c/cmd/changefeed.toml
- 2: Log start-ts in log when TiCDC MySQL sink fails to execute DML/DDL tiflow#6460
🏁 Script executed:
#!/bin/bash
set -eu
base='https://raw.githubusercontent.com/pingcap/tiflow/master/'
for f in \
pkg/sink/codec/debezium/encoder.go \
pkg/sink/codec/debezium/helper.go \
pkg/sink/codec/debezium/codec.go \
pkg/sink/codec/debezium/decoder.go \
pkg/sink/codec/common/config.go
do
printf '\n--- %s ---\n' "$f"
curl -fsSL "$base$f" | rg -n -C 8 'start[_-]?ts|StartTs|uint64|int64|Number|schema' || true
done
printf '\n--- available v8.5 tags ---\n'
curl -fsSL 'https://api.github.com/repos/pingcap/tiflow/tags?per_page=100' |
jq -r '.[].name' | rg '^v8\.5|^release-8\.5' | head -50Repository: pingcap/docs
Length of output: 36218
🏁 Script executed:
#!/bin/bash
set -eu
base='https://raw.githubusercontent.com/pingcap/tiflow/master/pkg/sink/codec/debezium/encoder.go'
printf '%s\n' '--- source schema and DML encoding ---'
curl -fsSL "$base" | sed -n '800,930p;980,1145p'
printf '%s\n' '--- refs containing v8.5.9 or v9.0 ---'
curl -fsSL 'https://api.github.com/repos/pingcap/tiflow/git/refs?per_page=100' |
jq -r '.[].ref' | rg 'v8\.5\.9|v9\.0|release-8\.5|release-9' | head -100
printf '%s\n' '--- feature references in upstream commit metadata ---'
curl -fsSL 'https://api.github.com/search/commits?q=repo%3Apingcap%2Ftiflow+include-start-ts' \
-H 'Accept: application/vnd.github+json' |
jq -r '.items[]? | [.sha, .commit.message] | `@tsv`' | head -20Repository: pingcap/docs
Length of output: 325
🏁 Script executed:
#!/bin/bash
set -eu
base='https://raw.githubusercontent.com/pingcap/tiflow/master/pkg/sink/codec/debezium/codec.go'
src=$(mktemp)
trap 'rm -f "$src"' EXIT
curl -fsSL "$base" -o "$src"
printf '%s\n' '--- source schema definition ---'
rg -n -C 35 'func \(c \*dbzCodec\) writeSourceSchema|func \(c \*dbzCodec\) EncodeRow|commit_ts' "$src" | head -260
printf '%s\n' '--- JSON writer imports and uint64 calls ---'
head -35 "$src"
printf '%s\n' '--- writer-related upstream paths ---'
curl -fsSL 'https://api.github.com/repos/pingcap/tiflow/git/trees/master?recursive=1' |
jq -r '.tree[] | select(.type == "blob") | .path' |
rg 'json|writer' | rg 'pkg/(util|sink)' | head -100Repository: pingcap/docs
Length of output: 13397
🏁 Script executed:
#!/bin/bash
set -eu
printf '%s\n' '--- source schema fields ---'
curl -fsSL 'https://raw.githubusercontent.com/pingcap/tiflow/master/pkg/sink/codec/debezium/codec.go' |
sed -n '844,970p'
printf '%s\n' '--- JSON writer uint64 implementation ---'
curl -fsSL 'https://raw.githubusercontent.com/pingcap/tiflow/master/pkg/util/json_writer.go' |
rg -n -C 12 'WriteUint64Field|WriteInt64Field|WriteAnyField'Repository: pingcap/docs
Length of output: 7689
🌐 Web query:
site:github.com/pingcap/tiflow "source.start_ts" OR "include-start-ts"
💡 Result:
The term source.start_ts (commonly referred to as start-ts or startTs in technical contexts, particularly within the TiKV/TiDB/TiCDC ecosystem) refers to the starting timestamp of a data replication stream, task, or transaction filter [1][2][3][4]. In the context of TiCDC (a Change Data Capture tool for TiDB), start-ts determines the point in time from which the changefeed begins capturing and replicating data changes [1][5]. Key characteristics include: 1. Function: It acts as a cursor for the replication process [3]. When creating a changefeed, specifying an start-ts instructs the system to start processing events from that specific transaction timestamp [1][5]. 2. Validation: To ensure data integrity, start-ts is typically validated against the Garbage Collection (GC) safepoint [1][6]. If a start-ts is older than the GC safepoint, the data has likely been cleaned up, and the changefeed will fail or encounter errors [5][6]. 3. Filtering: In some configurations, transaction filtering is performed based on the start-ts, allowing users to ignore specific transactions by listing their timestamps [7][4]. While there is no standard global parameter specifically named source.start_ts with an include-start-ts sub-option in general computing, variations of this naming pattern appear in configuration files, CLI flags, and API definitions where timestamps are required to initiate tasks [8][9][3][7]. If you are encountering include-start-ts in a specific piece of software, it likely functions as a boolean flag or configuration property to determine whether the event associated with the start_ts itself should be included in the processed stream [7].
Citations:
- 1: start-ts is not verified when creating the changefeed tiflow#10499
- 2: cmd: check changefeed start-ts when creating or resuming tiflow#1497
- 3: https://github.com/pingcap/tiflow/blob/8879687f/cdc/owner/changefeed.go
- 4: filter: using startts to filter txn tiflow#589
- 5: cdc cli changefeed create --start-ts does not work properly and may lead to data loss tiflow#2303
- 6: Can not create changefeed when start ts is less than current ts - gcttl tiflow#1839
- 7: https://github.com/pingcap/tiflow/blob/7c3c2336f98153326912f3cf6ea2fbb7bcc4a20c/cmd/changefeed.toml
- 8: https://github.com/pingcap/tiflow/blob/master/dm/openapi/spec/dm.yaml
- 9: https://github.com/pingcap/tiflow/blob/release-7.5/dm/openapi/spec/dm.yaml
🌐 Web query:
TiDB TSO uint64 signed int64 maximum range physical logical bits
💡 Result:
In TiDB, the Timestamp Oracle (TSO) provides a 64-bit timestamp that functions as a monotonically increasing identifier [1][2]. While the underlying storage or transport might sometimes involve signed 64-bit integers (int64) in specific programming contexts [3][4], the TSO is conceptually and structurally treated as a 64-bit unsigned integer (uint64) [5][4]. Structure of the 64-bit TSO: - Physical Component: The upper 46 bits represent the physical timestamp, defined as the number of milliseconds elapsed since the Unix epoch (1 January 1970) [1][4]. - Logical Component: The lower 18 bits represent the logical counter [1][2]. This counter allows TiDB to generate up to 2^18 (262,144) unique timestamps within a single millisecond [3][4]. Technical Details: - Bitwise Composition: A TSO is composed by shifting the physical timestamp 18 bits to the left and adding the logical counter: (physical << 18) | logical [5][4]. - Capacity: The 46-bit physical field provides a range of approximately 2,230 years from the Unix epoch, which is sufficient for all practical requirements [4]. The 18-bit logical field ensures that if multiple requests occur within the same millisecond, they can be assigned distinct, ordered timestamps by incrementing the logical counter [1][3]. - Overflow Handling: If the logical counter exceeds its maximum capacity (2^18 - 1), the system waits for the physical clock to advance to the next millisecond to ensure the timestamp remains strictly increasing [3][2]. In practice, developers often use bitwise operations to parse these components—specifically, right-shifting the 64-bit value by 18 bits to extract the physical time and using a bitwise AND mask (0x3FFFF) to extract the logical counter [1][6][4]. Although language-specific implementations (such as in Go or Rust) may use int64 or u64 types, the bitwise structure remains consistent across the architecture [5][3][4].
Citations:
- 1: https://docs.pingcap.com/tidb/stable/tso/
- 2: https://www.pingcap.com/blog/how-an-open-source-distributed-newsql-database-delivers-time-services/
- 3: https://github.com/tikv/pd/wiki/Timestamp-Oracle
- 4: https://jzleetcode.github.io/posts/design-how-tidb-tso-works/
- 5: https://tikv.github.io/doc/src/txn_types/timestamp.rs.html
- 6: https://docs.pingcap.com/tidb/dev/tso/
Document the signed int64 range for source.start_ts.
start_ts is documented as a uint64 TSO, but its schema is documented as signed int64. An int64 schema supports only values from 0 through 9223372036854775807; values above this range can overflow or be rejected. Distinguish the JSON number representation from the schema type, and state the supported range or use a representation that preserves the full uint64 domain.
Source: Path instructions
| When this option is enabled: | ||
|
|
||
| - DML value messages add integer `source.start_ts` next to `commit_ts`, and the JSON schema declares the field as `int64`. | ||
| - DDL events, WATERMARK/checkpoint events, key messages, and Debezium Avro are unchanged. Setting this option with the Debezium Avro protocol is rejected. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Use Avro protocol consistently.
The supplied protocol lists define avro and debezium as separate values. “Debezium Avro” is not the established protocol name and can confuse configuration users.
ticdc/ticdc-debezium.md#L56-L56: refer to the rejected format as theAvro protocol.ticdc/ticdc-changefeed-config.md#L360-L360: refer to the rejected format as theAvro protocol.ticdc/ticdc-sink-to-kafka.md#L86-L86: refer to the rejected format as theAvro protocol.
As per path instructions: preserve established terminology in Markdown.
📍 Affects 3 files
ticdc/ticdc-debezium.md#L56-L56(this comment)ticdc/ticdc-changefeed-config.md#L360-L360ticdc/ticdc-sink-to-kafka.md#L86-L86
Source: Path instructions
What is changed, added or deleted? (Required)
Document the optional TiCDC Debezium JSON
source.start_tsfield (new in v8.5.9). Downstream consumers can use the original PD TSO of the source transaction for exact-once or transaction grouping. The option is disabled by default.debezium-include-start-ts, TOML[sink.debezium] include-start-ts, URI precedence, DML-only scope, Avro rejection, rollback, and uint64 TSO precision (do not parse as JavaScriptNumber/ float64).include_start_tsfield, and the DMLpayload.source.start_tstable row.Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions.
What is the related PR or file link(s)?
AI agent involvement
Do your changes match any of the following descriptions?
Summary by CodeRabbit
include-start-tssetting for Debezium JSON changefeed messages.source.start_tsto DML events.