Skip to content

server: fence capture writes with etcd and P2P leases - #6092

Draft
asddongmen wants to merge 19 commits into
pingcap:masterfrom
asddongmen:0826-capture-write-lease
Draft

server: fence capture writes with etcd and P2P leases#6092
asddongmen wants to merge 19 commits into
pingcap:masterfrom
asddongmen:0826-capture-write-lease

Conversation

@asddongmen

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: ref #6045

A network-isolated capture can keep writing after the surviving coordinator schedules replacement dispatchers. This creates an old/new writer overlap and can leave a MySQL sink inconsistent even after the checkpoint advances.

What is changed and how it works?

  • Add a capture-wide write gate backed by fresh etcd and coordinator-issued P2P proofs.
  • Renew P2P proof through capability-negotiated node heartbeat responses, with a remote witness for a coordinator colocated with its capture.
  • Fence downstream writes when either proof expires; confirmed etcd session loss still exits the capture.
  • Check the gate at MySQL DML execution and expose lease/admission metrics and deterministic failpoints.
  • Extend capture removal grace to 10 seconds.

Already-admitted MySQL transactions remain outside the strict guarantee: the gate cannot revoke a COMMIT already accepted by the downstream.

Check List

Tests

  • Unit test
  • Integration test: three-capture MySQL chaos matrix and admitted-COMMIT boundary passed in TCMS plan 8228411.
  • Manual test: formal 20-run stress plan 8228428 is running.

Questions

Will it cause performance regression or break compatibility?

The DML admission check is an in-memory gate. New heartbeat traffic runs every 500 ms. Enforcement is negotiated per node, so old nodes that do not advertise the capability keep legacy behavior during rolling upgrade.

Do you need to update user documentation, design documentation or monitoring documentation?

No user documentation change. New metrics expose write-lease state; the design remains an internal review draft until finalized.

Release note

Reduce the risk of duplicate MySQL sink writes during TiCDC capture network isolation by fencing writes with etcd and P2P leases.

@ti-chi-bot

ti-chi-bot Bot commented Aug 26, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot ti-chi-bot Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-cherry-pick-release-nextgen-202603 Should cherry pick this PR to release-nextgen-202603 branch. labels Aug 26, 2026
@ti-chi-bot

ti-chi-bot Bot commented Aug 26, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tenfyzhong for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot ti-chi-bot Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-cherry-pick-release-nextgen-202603 Should cherry pick this PR to release-nextgen-202603 branch. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant