Skip to content

ticdc: document Debezium source.start_ts option #23614

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ekexium wants to merge 1 commit into
base: master
Choose a base branch
from
+52 −0
Open

ticdc: document Debezium source.start_ts option #23614

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions ticdc/ticdc-changefeed-config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -353,6 +353,14 @@ The following configuration parameters control the sending behavior of bootstrap
- Controls whether to output the value before the row data changes. The default value is true. When it is disabled, the `UPDATE` event does not output the "before" field.
- Default value: `true`

##### `include-start-ts` <span class="version-mark">New in v8.5.9</span>

- Controls whether Debezium JSON DML messages include `source.start_ts` (the original PD TSO of the source transaction).
- Default value: `false`
- This parameter takes effect only when the sink type is MQ and the output protocol is Debezium JSON. Setting it with Debezium Avro is rejected.
- You can also set the equivalent URI parameter `debezium-include-start-ts`. An explicit URI value takes precedence over this configuration item, including `false` overriding `true`.
- For the message format and consumer precision requirements, see [TiCDC Debezium Protocol](/ticdc/ticdc-debezium.md#include-the-transaction-start-tso).

### consistent

Specifies the replication consistency configurations for a changefeed when using the redo log. For more information, see [Eventually consistent replication in disaster scenarios](/ticdc/ticdc-sink-to-mysql.md#eventually-consistent-replication-in-disaster-scenarios).
Expand Down
42 changes: 42 additions & 0 deletions ticdc/ticdc-debezium.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,47 @@ The Debezium output format contains the schema information of the current row so

In addition, the original Debezium format does not include important fields such as the unique transaction identifier of the `CommitTS` in TiDB. To ensure data integrity, TiCDC adds two fields, `CommitTs` and `ClusterID`, to the Debezium format to identify the relevant information of TiDB data changes.

### Include the transaction start TSO <span class="version-mark">New in v8.5.9</span>

By default, Debezium JSON DML messages include `source.commit_ts` but not the transaction start TSO. You can optionally include `source.start_ts` (the original PD TSO when the source transaction started) on DML row events only. This option is disabled by default.

You can enable it in either of the following ways:

- In `sink-uri`:

```
kafka://127.0.0.1:9092/topic-name?protocol=debezium&debezium-include-start-ts=true
```

- In the changefeed configuration file:

```toml
[sink.debezium]
include-start-ts = true
```

An explicit URI value takes precedence over the configuration file, including `debezium-include-start-ts=false` overriding `include-start-ts = true`.

When this option is enabled:

- DML value messages add integer `source.start_ts` next to `commit_ts`, and the JSON schema declares the field as `int64`.

@coderabbitai coderabbitai Bot Aug 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

#!/bin/bash
# Expected: encoder, schema, and API definitions agree on signed/unsigned and JSON representation.
rg -n -C 8 'include[-_]start[-_]ts|includeStartTs|start_ts|startTs|uint64|int64' . \
  --glob '*.go' --glob '*.proto' --glob '*.json' --glob '*.yaml' --glob '*.toml'

Repository: pingcap/docs

Length of output: 150

🏁 Script executed:

#!/bin/bash
set -eu
printf '%s\n' '--- repository convention files ---'
find /tmp/coderabbit-repo-knowledge/pingcap-docs-9c73f70c -maxdepth 2 -type f -print | sort
printf '%s\n' '--- reviewed document ---'
sed -n '30,75p' ticdc/ticdc-debezium.md
printf '%s\n' '--- relevant tracked files ---'
git ls-files | rg '(^|/)(ticdc|resources|scripts)/|debezium|cdc|protocol' | head -200
printf '%s\n' '--- documentation references ---'
rg -n -C 4 'start_ts|startTs|include[-_]start[-_]ts|includeStartTs|uint64|int64|Number|Debezium Avro|protocol' ticdc resources README.md 2>/dev/null | head -300

Repository: pingcap/docs

Length of output: 36589

🏁 Script executed:

#!/bin/bash
set -eu
printf '%s\n' '--- repository-wide convention ---'
cat /tmp/coderabbit-repo-knowledge/pingcap-docs-9c73f70c/conventions/repo-wide.md
printf '%s\n' '--- exact document context ---'
sed -n '1,70p' ticdc/ticdc-debezium.md
printf '%s\n' '--- local option references ---'
rg -n -C 8 'include-start-ts|debezium-include-start-ts|source\.start_ts' ticdc

Repository: pingcap/docs

Length of output: 22621

🌐 Web query:

pingcap tiflow include-start-ts debezium source.start_ts uint64 int64 implementation

💡 Result:

In TiCDC, the start-ts of a changefeed is a Timestamp Oracle (TSO) value [1][2]. When using the Debezium protocol for TiCDC, the protocol output includes metadata fields in the payload, such as payload.source.commit_ts [3]. Regarding your query about include-start-ts, source.start_ts, and type mappings: 1. Start-ts: This is a uint64 value representing the starting TSO in the upstream TiDB cluster [1][4]. When creating or resuming a changefeed, this value is used to specify the point from which TiCDC begins pulling data [5][6]. If not specified (or set to 0), TiCDC fetches a current TSO to start the task [1][2]. 2. Debezium protocol output: TiCDC's Debezium protocol output is designed to be compatible with downstream systems that expect the standard Debezium format, while adding specific fields to ensure data integrity in TiDB [3]. Specifically, TiCDC adds CommitTs and ClusterID to the payload's source section [3]. The field payload.source.commit_ts is included as a Number type [3]. 3. Data types: TiDB timestamps (TSOs) are 64-bit unsigned integers (uint64) [1][7]. When these values are serialized into JSON for the Debezium protocol, they are typically represented as numbers [3]. Downstream systems or custom consumers reading these JSON messages should handle these as 64-bit integers to prevent precision loss or overflow, as standard JSON numbers are often parsed as 64-bit floating-point numbers in some languages [3]. There is no standard configuration parameter named include-start-ts or source.start_ts in the TiCDC Debezium sink configuration [5][6]. The core parameter for defining the start point is the --start-ts flag used during changefeed creation [5][6]. If you are encountering references to source.start_ts in your integration, it is likely a custom mapping or a reference to the CommitTs field which serves the purpose of identifying the transaction order [3].

Citations:

🏁 Script executed:

#!/bin/bash
set -eu
tmp=$(mktemp)
trap 'rm -f "$tmp"' EXIT
curl -fsSL 'https://api.github.com/repos/pingcap/tiflow/git/trees/master?recursive=1' -o "$tmp"
printf '%s\n' '--- matching upstream paths ---'
jq -r '.tree[] | select(.path | test("debezium|codec|sink")) | .path' "$tmp" | rg -i 'debezium|codec' | head -200
printf '%s\n' '--- source paths containing likely option names ---'
jq -r '.tree[] | select(.type == "blob") | .path' "$tmp" | rg -i '\.(go|proto|json|yaml|toml)$' | head -300

Repository: pingcap/docs

Length of output: 19063

🌐 Web query:

site:github.com/pingcap/tiflow "debezium-include-start-ts"

💡 Result:

There is no configuration parameter named debezium-include-start-ts or include.start.ts in the standard Debezium connector configuration. Based on the technical context of tools like TiCDC (a Change Data Capture tool that supports the Debezium output format), the term "start-ts" refers to a transaction start timestamp, which is a core concept in TiDB-based replication systems (such as ignore-txn-start-ts in TiCDC configuration files) [1]. If you are encountering a reference to start-ts in relation to Debezium, it is likely appearing in one of the following contexts: 1. TiCDC Configuration: When configuring a changefeed to output data in the Debezium protocol, users manage transaction timestamps using settings like ignore-txn-start-ts within the changefeed's TOML configuration file [1]. 2. Troubleshooting and Logs: System logs for data replication often include the start-ts of transactions when operations fail or to track progress, but this is not a connector-level "include" flag [2]. If you are looking for ways to include timestamps in Debezium events, Debezium naturally includes source-side timestamps (such as source.ts_ms or ts_ms) in its standard JSON or Avro message formats, which do not require a specific "include" configuration parameter.

Citations:

🏁 Script executed:

#!/bin/bash
set -eu
base='https://raw.githubusercontent.com/pingcap/tiflow/master/'
for f in \
  pkg/sink/codec/debezium/encoder.go \
  pkg/sink/codec/debezium/helper.go \
  pkg/sink/codec/debezium/codec.go \
  pkg/sink/codec/debezium/decoder.go \
  pkg/sink/codec/common/config.go
do
  printf '\n--- %s ---\n' "$f"
  curl -fsSL "$base$f" | rg -n -C 8 'start[_-]?ts|StartTs|uint64|int64|Number|schema' || true
done
printf '\n--- available v8.5 tags ---\n'
curl -fsSL 'https://api.github.com/repos/pingcap/tiflow/tags?per_page=100' |
  jq -r '.[].name' | rg '^v8\.5|^release-8\.5' | head -50

Repository: pingcap/docs

Length of output: 36218

🏁 Script executed:

#!/bin/bash
set -eu
base='https://raw.githubusercontent.com/pingcap/tiflow/master/pkg/sink/codec/debezium/encoder.go'
printf '%s\n' '--- source schema and DML encoding ---'
curl -fsSL "$base" | sed -n '800,930p;980,1145p'
printf '%s\n' '--- refs containing v8.5.9 or v9.0 ---'
curl -fsSL 'https://api.github.com/repos/pingcap/tiflow/git/refs?per_page=100' |
  jq -r '.[].ref' | rg 'v8\.5\.9|v9\.0|release-8\.5|release-9' | head -100
printf '%s\n' '--- feature references in upstream commit metadata ---'
curl -fsSL 'https://api.github.com/search/commits?q=repo%3Apingcap%2Ftiflow+include-start-ts' \
  -H 'Accept: application/vnd.github+json' |
  jq -r '.items[]? | [.sha, .commit.message] | `@tsv`' | head -20

Repository: pingcap/docs

Length of output: 325

🏁 Script executed:

#!/bin/bash
set -eu
base='https://raw.githubusercontent.com/pingcap/tiflow/master/pkg/sink/codec/debezium/codec.go'
src=$(mktemp)
trap 'rm -f "$src"' EXIT
curl -fsSL "$base" -o "$src"
printf '%s\n' '--- source schema definition ---'
rg -n -C 35 'func \(c \*dbzCodec\) writeSourceSchema|func \(c \*dbzCodec\) EncodeRow|commit_ts' "$src" | head -260
printf '%s\n' '--- JSON writer imports and uint64 calls ---'
head -35 "$src"
printf '%s\n' '--- writer-related upstream paths ---'
curl -fsSL 'https://api.github.com/repos/pingcap/tiflow/git/trees/master?recursive=1' |
  jq -r '.tree[] | select(.type == "blob") | .path' |
  rg 'json|writer' | rg 'pkg/(util|sink)' | head -100

Repository: pingcap/docs

Length of output: 13397

🏁 Script executed:

#!/bin/bash
set -eu
printf '%s\n' '--- source schema fields ---'
curl -fsSL 'https://raw.githubusercontent.com/pingcap/tiflow/master/pkg/sink/codec/debezium/codec.go' |
  sed -n '844,970p'
printf '%s\n' '--- JSON writer uint64 implementation ---'
curl -fsSL 'https://raw.githubusercontent.com/pingcap/tiflow/master/pkg/util/json_writer.go' |
  rg -n -C 12 'WriteUint64Field|WriteInt64Field|WriteAnyField'

Repository: pingcap/docs

Length of output: 7689

🌐 Web query:

site:github.com/pingcap/tiflow "source.start_ts" OR "include-start-ts"

💡 Result:

The term source.start_ts (commonly referred to as start-ts or startTs in technical contexts, particularly within the TiKV/TiDB/TiCDC ecosystem) refers to the starting timestamp of a data replication stream, task, or transaction filter [1][2][3][4]. In the context of TiCDC (a Change Data Capture tool for TiDB), start-ts determines the point in time from which the changefeed begins capturing and replicating data changes [1][5]. Key characteristics include: 1. Function: It acts as a cursor for the replication process [3]. When creating a changefeed, specifying an start-ts instructs the system to start processing events from that specific transaction timestamp [1][5]. 2. Validation: To ensure data integrity, start-ts is typically validated against the Garbage Collection (GC) safepoint [1][6]. If a start-ts is older than the GC safepoint, the data has likely been cleaned up, and the changefeed will fail or encounter errors [5][6]. 3. Filtering: In some configurations, transaction filtering is performed based on the start-ts, allowing users to ignore specific transactions by listing their timestamps [7][4]. While there is no standard global parameter specifically named source.start_ts with an include-start-ts sub-option in general computing, variations of this naming pattern appear in configuration files, CLI flags, and API definitions where timestamps are required to initiate tasks [8][9][3][7]. If you are encountering include-start-ts in a specific piece of software, it likely functions as a boolean flag or configuration property to determine whether the event associated with the start_ts itself should be included in the processed stream [7].

Citations:

🌐 Web query:

TiDB TSO uint64 signed int64 maximum range physical logical bits

💡 Result:

In TiDB, the Timestamp Oracle (TSO) provides a 64-bit timestamp that functions as a monotonically increasing identifier [1][2]. While the underlying storage or transport might sometimes involve signed 64-bit integers (int64) in specific programming contexts [3][4], the TSO is conceptually and structurally treated as a 64-bit unsigned integer (uint64) [5][4]. Structure of the 64-bit TSO: - Physical Component: The upper 46 bits represent the physical timestamp, defined as the number of milliseconds elapsed since the Unix epoch (1 January 1970) [1][4]. - Logical Component: The lower 18 bits represent the logical counter [1][2]. This counter allows TiDB to generate up to 2^18 (262,144) unique timestamps within a single millisecond [3][4]. Technical Details: - Bitwise Composition: A TSO is composed by shifting the physical timestamp 18 bits to the left and adding the logical counter: (physical << 18) | logical [5][4]. - Capacity: The 46-bit physical field provides a range of approximately 2,230 years from the Unix epoch, which is sufficient for all practical requirements [4]. The 18-bit logical field ensures that if multiple requests occur within the same millisecond, they can be assigned distinct, ordered timestamps by incrementing the logical counter [1][3]. - Overflow Handling: If the logical counter exceeds its maximum capacity (2^18 - 1), the system waits for the physical clock to advance to the next millisecond to ensure the timestamp remains strictly increasing [3][2]. In practice, developers often use bitwise operations to parse these components—specifically, right-shifting the 64-bit value by 18 bits to extract the physical time and using a bitwise AND mask (0x3FFFF) to extract the logical counter [1][6][4]. Although language-specific implementations (such as in Go or Rust) may use int64 or u64 types, the bitwise structure remains consistent across the architecture [5][3][4].

Citations:

Document the signed int64 range for source.start_ts.

start_ts is documented as a uint64 TSO, but its schema is documented as signed int64. An int64 schema supports only values from 0 through 9223372036854775807; values above this range can overflow or be rejected. Distinguish the JSON number representation from the schema type, and state the supported range or use a representation that preserves the full uint64 domain.

Source: Path instructions

- DDL events, WATERMARK/checkpoint events, key messages, and Debezium Avro are unchanged. Setting this option with the Debezium Avro protocol is rejected.

@coderabbitai coderabbitai Bot Aug 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Use Avro protocol consistently.

The supplied protocol lists define avro and debezium as separate values. “Debezium Avro” is not the established protocol name and can confuse configuration users.

  • ticdc/ticdc-debezium.md#L56-L56: refer to the rejected format as the Avro protocol.
  • ticdc/ticdc-changefeed-config.md#L360-L360: refer to the rejected format as the Avro protocol.
  • ticdc/ticdc-sink-to-kafka.md#L86-L86: refer to the rejected format as the Avro protocol.

As per path instructions: preserve established terminology in Markdown.

📍 Affects 3 files
  • ticdc/ticdc-debezium.md#L56-L56 (this comment)
  • ticdc/ticdc-changefeed-config.md#L360-L360
  • ticdc/ticdc-sink-to-kafka.md#L86-L86

Source: Path instructions

- To roll back, disable the option. Messages produced while it is off stay byte-compatible with the previous format.

> **Note:**
>
> `start_ts` is the original uint64 PD TSO, not a millisecond timestamp. Consumers must treat it as a 64-bit integer or a decimal string. Do not parse it as a JavaScript `Number` or IEEE-754 `float64`, which cannot represent an 18-digit TSO exactly.

When the option is enabled, the `source` block looks like the following:

```json
"source": {
"commit_ts": 447507027004751877,
"start_ts": 447507027004751800,
"cluster_id": "default"
}
```

## Message format definition

This section describes the message formats of DDL events, DML events and WATERMARK events.
Expand Down Expand Up @@ -572,6 +613,7 @@ The key fields of the preceding JSON data are explained as follows:
| `payload.before` | JSON | The data value before the change event of a statement. For `"c"` events, the value of the `before` field is `null`. |
| `payload.after` | JSON | The data value after the change event of a statement. For `"d"` events, the value of the `after` field is `null`. |
| `payload.source.commit_ts` | Number | The `CommitTs` value of the event. |
| `payload.source.start_ts` | Number | The start TSO of the source transaction. Present only when `debezium-include-start-ts` or `[sink.debezium] include-start-ts` is enabled. Original uint64 PD TSO, not a millisecond timestamp. |
| `payload.source.db` | String | The name of the database where the event occurs. |
| `payload.source.table` | String | The name of the table where the event occurs. |
| `schema.fields` | JSON | The type information of each field in the payload, including the schema information of the row data before and after the change. |
Expand Down
1 change: 1 addition & 0 deletions ticdc/ticdc-open-api-v2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -373,6 +373,7 @@ The `sink.csv` parameters are described as follows:
| Parameter name | Description |
|:-------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `output_old_value` | `BOOLEAN` type. It controls whether to output the value before the row data changes. The default value is true. When it is disabled, the UPDATE event does not output the "before" field. |
| `include_start_ts` | `BOOLEAN` type. New in v8.5.9. It controls whether Debezium JSON DML messages include `source.start_ts` (the original PD TSO of the source transaction). The default value is `false`. |

### Example

Expand Down
1 change: 1 addition & 0 deletions ticdc/ticdc-sink-to-kafka.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ The following are descriptions of sink URI parameters and values that can be con
| `compression` | The compression algorithm used when sending messages (value options are `none`, `lz4`, `gzip`, `snappy`, and `zstd`; `none` by default). Note that the Snappy compressed file must be in the [official Snappy format](https://github.com/google/snappy). Other variants of Snappy compression are not supported.|
| `auto-create-topic` | Determines whether TiCDC creates the topic automatically when the `topic-name` passed in does not exist in the Kafka cluster (optional, `true` by default). |
| `enable-tidb-extension` | Optional. `false` by default. When the output protocol is `canal-json`, if the value is `true`, TiCDC sends [WATERMARK events](/ticdc/ticdc-canal-json.md#watermark-event) and adds the [TiDB extension field](/ticdc/ticdc-canal-json.md#tidb-extension-field) to Kafka messages. From v6.1.0, this parameter is also applicable to the `avro` protocol. If the value is `true`, TiCDC adds [three TiDB extension fields](/ticdc/ticdc-avro-protocol.md#tidb-extension-fields) to the Kafka message. |
| `debezium-include-start-ts` | Optional. New in v8.5.9. `false` by default. Takes effect only when `protocol` is `debezium`. If the value is `true`, TiCDC adds `source.start_ts` (the original PD TSO of the source transaction) to Debezium JSON DML messages. An explicit URI value takes precedence over `[sink.debezium] include-start-ts` in the configuration file. This option is rejected for Debezium Avro. For details, see [TiCDC Debezium Protocol](/ticdc/ticdc-debezium.md#include-the-transaction-start-tso). |
| `max-batch-size` | New in v4.0.9. If the message protocol supports outputting multiple data changes to one Kafka message, this parameter specifies the maximum number of data changes in one Kafka message. It currently takes effect only when Kafka's `protocol` is `open-protocol` (optional, `16` by default). |
| `enable-tls` | Whether to use TLS to connect to the downstream Kafka instance (optional, `false` by default). |
| `ca` | The path of the CA certificate file needed to connect to the downstream Kafka instance (optional). |
Expand Down
Loading