Skip to content

coordinator: Support resume successfully when status have previous errors#3954

Merged
ti-chi-bot[bot] merged 4 commits intopingcap:masterfrom
hongyunyan:0108-resume
Jan 12, 2026
Merged

coordinator: Support resume successfully when status have previous errors#3954
ti-chi-bot[bot] merged 4 commits intopingcap:masterfrom
hongyunyan:0108-resume

Conversation

@hongyunyan
Copy link
Copy Markdown
Collaborator

@hongyunyan hongyunyan commented Jan 8, 2026

What problem does this PR solve?

Issue Number: close #3953

What is changed and how it works?

This pull request significantly enhances the robustness of changefeed resume operations within the coordinator. It addresses a critical issue where changefeeds might fail to restart or immediately re-enter a problematic state due to lingering error information or an improperly initialized backoff mechanism from prior failures. By ensuring that resume attempts start with a clean, error-free status and a correctly reset internal state, the changes aim to provide a more reliable and predictable user experience for managing changefeeds.

Highlights

  • Improved Changefeed Resume Logic: Changefeeds can now be successfully resumed even if their status previously contained errors. The system will no longer be blocked by stale error states during a resume operation.
  • Error-Free Status for Resume: A new method, GetStatusForResume, has been introduced which provides a deep copy of the changefeed status but explicitly excludes any historical running errors, ensuring a clean state for resume attempts.
  • Backoff Mechanism Reset: When a changefeed is resumed, its internal backoff mechanism is now properly reset. This includes updating the checkpoint baseline and clearing any in-progress restarting markers, preventing the changefeed from being immediately flagged as 'stuck' based on its previous failed state.
  • Enhanced Test Coverage: New unit and integration tests have been added to validate that changefeeds can resume correctly despite stale errors and that the backoff checkpoint baseline is reset as expected.

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Please refer to [Release Notes Language Style Guide](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/release-notes-style-guide.html) to write a quality release note.

If you don't think this PR needs a release note then fill it with `None`.

@ti-chi-bot ti-chi-bot Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jan 8, 2026
@gemini-code-assist
Copy link
Copy Markdown

Summary of Changes

Hello @hongyunyan, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the robustness of changefeed resume operations within the coordinator. It addresses a critical issue where changefeeds might fail to restart or immediately re-enter a problematic state due to lingering error information or an improperly initialized backoff mechanism from prior failures. By ensuring that resume attempts start with a clean, error-free status and a correctly reset internal state, the changes aim to provide a more reliable and predictable user experience for managing changefeeds.

Highlights

  • Improved Changefeed Resume Logic: Changefeeds can now be successfully resumed even if their status previously contained errors. The system will no longer be blocked by stale error states during a resume operation.
  • Error-Free Status for Resume: A new method, GetStatusForResume, has been introduced which provides a deep copy of the changefeed status but explicitly excludes any historical running errors, ensuring a clean state for resume attempts.
  • Backoff Mechanism Reset: When a changefeed is resumed, its internal backoff mechanism is now properly reset. This includes updating the checkpoint baseline and clearing any in-progress restarting markers, preventing the changefeed from being immediately flagged as 'stuck' based on its previous failed state.
  • Enhanced Test Coverage: New unit and integration tests have been added to validate that changefeeds can resume correctly despite stale errors and that the backoff checkpoint baseline is reset as expected.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Copy Markdown

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request aims to allow changefeeds to resume successfully even if they have previous errors in their status. The changes correctly implement this by creating a new function GetStatusForResume that provides a clean status without old errors. Additionally, there are good improvements in GC safepoint handling on resume failure and in resetting the backoff logic to prevent incorrect "stuck" state detection. However, I've found a critical issue where the checkpoint timestamp is not correctly updated when resuming a changefeed, which could lead to the changefeed starting from an incorrect position.

Comment thread coordinator/controller.go
@ti-chi-bot ti-chi-bot Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jan 9, 2026
@ti-chi-bot ti-chi-bot Bot added needs-1-more-lgtm Indicates a PR needs 1 more LGTM. approved labels Jan 12, 2026
@ti-chi-bot
Copy link
Copy Markdown

ti-chi-bot Bot commented Jan 12, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 3AceShowHand, wk989898

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:
  • OWNERS [3AceShowHand,wk989898]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jan 12, 2026
@ti-chi-bot
Copy link
Copy Markdown

ti-chi-bot Bot commented Jan 12, 2026

[LGTM Timeline notifier]

Timeline:

  • 2026-01-12 07:27:56.862168329 +0000 UTC m=+256120.924033238: ☑️ agreed by 3AceShowHand.
  • 2026-01-12 08:01:13.824145416 +0000 UTC m=+258117.886010326: ☑️ agreed by wk989898.

@ti-chi-bot ti-chi-bot Bot merged commit 0d38515 into pingcap:master Jan 12, 2026
19 checks passed
a-cong pushed a commit to a-cong/ticdc that referenced this pull request Feb 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved lgtm release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Changefeed resume not successfully if changefeed have previous errs

3 participants