Skip to content

eventservice: fix broker quota release and message validation#5643

Open
lidezhu wants to merge 2 commits into
masterfrom
ldz/fix-event-broker0712
Open

eventservice: fix broker quota release and message validation#5643
lidezhu wants to merge 2 commits into
masterfrom
ldz/fix-event-broker0712

Conversation

@lidezhu

@lidezhu lidezhu commented Jul 12, 2026

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: close #5642

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Questions

Will it cause performance regression or break compatibility?
Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Please refer to [Release Notes Language Style Guide](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/release-notes-style-guide.html) to write a quality release note.

If you don't think this PR needs a release note then fill it with `None`.

Summary by CodeRabbit

  • Bug Fixes

    • Fixed changefeed scan quota handling so allocated capacity is properly returned after scan failures, interruptions, and partial scans.
    • Prevented invalid heartbeat and congestion-control messages from causing processing errors.
    • Corrected dispatcher metrics to include table-trigger dispatchers.
  • Tests

    • Added coverage for quota recovery, scan errors, invalid messages, and dispatcher metric tracking.

@ti-chi-bot

ti-chi-bot Bot commented Jul 12, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@ti-chi-bot ti-chi-bot Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Jul 12, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 12, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign wlwilliamx for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Jul 12, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e9281156-ab8a-4fcf-a49d-07641150ae6c

📥 Commits

Reviewing files that changed from the base of the PR and between 1971d2b and 47a25dc.

📒 Files selected for processing (5)
  • pkg/eventservice/event_broker.go
  • pkg/eventservice/event_broker_test.go
  • pkg/eventservice/event_service.go
  • pkg/eventservice/event_service_test.go
  • pkg/eventservice/test_helper.go

📝 Walkthrough

Walkthrough

Eventservice scan quota cleanup now covers early exits and scan failures, table-trigger dispatcher metrics remain balanced, and malformed heartbeat or congestion-control messages are ignored safely after logging.

Changes

Eventservice corrections

Layer / File(s) Summary
Scan quota release and failure coverage
pkg/eventservice/event_broker.go, pkg/eventservice/event_broker_test.go, pkg/eventservice/test_helper.go
doScan releases allocated quota on insufficient quota, scan errors, invalid scans, and partial scans. Tests cover quota restoration and injected table-info failures.
Table-trigger dispatcher gauge accounting
pkg/eventservice/event_broker.go, pkg/eventservice/event_broker_test.go
Table-trigger dispatcher registration increments the dispatcher gauge, and tests verify the gauge returns to baseline after removal.
Invalid message payload handling
pkg/eventservice/event_service.go, pkg/eventservice/event_service_test.go
Heartbeat and congestion-control messages with invalid payload lengths return early, with tests covering nil payloads.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested labels: lgtm, approved

Suggested reviewers: asddongmen, wk989898, flowbehappy

Poem

I’m a rabbit with quotas tucked neat,
No leaked bytes beneath my feet.
Gauges hop in balanced rows,
Bad messages meet quick “no.”
Tests cheer: “The burrow’s bright!” 🐇

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The template sections are present, but the change summary, checklist details, answers, and release note are left blank or generic. Fill in what changed, complete the test checklist and questions, and replace the release-note placeholder with a real note or None.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title matches the main fixes: quota release and malformed message handling in eventservice.
Linked Issues check ✅ Passed The PR addresses all three linked objectives: quota release, dispatcher gauge balance, and safe invalid-message handling.
Out of Scope Changes check ✅ Passed The added tests and mock support are directly tied to the stated fixes and no unrelated code changes stand out.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ldz/fix-event-broker0712

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@ti-chi-bot ti-chi-bot Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jul 12, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces fixes to the event service, including releasing memory quota when skipping scans due to insufficient quota, tracking table trigger dispatchers in metrics, and handling invalid message payloads by returning early. Corresponding unit tests have been added. The review feedback points out a potential quota leak if scanning fails later in the process, and warns about unsafe type assertions on message payloads that could lead to panics, suggesting safer 'comma-ok' type assertions instead.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

}

if uint64(sl.maxDMLBytes) > task.availableMemoryQuota.Load() {
releaseQuota(available, uint64(sl.maxDMLBytes))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

While releasing the allocated quota here is necessary, a similar quota leak can occur if scanner.scan fails further down in this function (around line 728). When scanner.scan returns an error, the remaining scannedBytes of the allocated quota is not fully released, which can lead to temporary quota exhaustion until the next congestion control update.

Consider fully releasing the remaining quota in the error handling block of scanner.scan as well:

if err != nil {
    if scannedBytes > 0 {
        releaseQuota(available, uint64(min(sl.maxDMLBytes, scannedBytes)))
    }
    log.Error("scan events failed", ...)
    return
}

Comment thread pkg/eventservice/event_service.go
Comment thread pkg/eventservice/event_service.go
@ti-chi-bot ti-chi-bot Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 12, 2026
@lidezhu lidezhu marked this pull request as ready for review July 12, 2026 02:03
@ti-chi-bot ti-chi-bot Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-note Denotes a PR that will be considered when it comes time to generate release notes. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

eventservice: fix quota leak, dispatcher gauge mismatch, and invalid message handling

1 participant