Skip to content

[AMORO-4034][Improvement]: MaintainerMetrics Unified interface and Report #4045

Open
czy006 wants to merge 2 commits intoapache:masterfrom
czy006:issues/amoro-4034
Open

[AMORO-4034][Improvement]: MaintainerMetrics Unified interface and Report #4045
czy006 wants to merge 2 commits intoapache:masterfrom
czy006:issues/amoro-4034

Conversation

@czy006
Copy link
Copy Markdown
Contributor

@czy006 czy006 commented Jan 16, 2026

[Improvement]: MaintainerMetrics Unified interface and Report (#4034)

Why are the changes needed?

Close #4034 .

Brief change log

  • MaintainerMetrics interface is abstracted to support a multi-lake architecture
  • Maintainer Metrics add other maintainer metrics interface: such as expireSnapshots/expireData/cleanDanglingDeleteFiles/autoCreateTags
  • MaintainerMetrics OrphanDataFiles method Change to a more general implementation

How was this patch tested?

  • Add some test cases that check the changes thoroughly including negative and positive cases if possible

  • Add screenshots for manual tests if appropriate

  • Run test locally before making a pull request

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Jan 16, 2026

Codecov Report

❌ Patch coverage is 57.95207% with 193 lines in your changes missing coverage. Please review.
✅ Project coverage is 30.03%. Comparing base (fe98a6c) to head (5bb09bd).
⚠️ Report is 15 commits behind head on master.

Files with missing lines Patch % Lines
...ats/iceberg/maintainer/IcebergTableMaintainer.java 23.29% 131 Missing and 4 partials ⚠️
...che/amoro/server/table/TableMaintainerMetrics.java 81.57% 41 Missing and 1 partial ⚠️
...zing/maintainer/DefaultTableMaintainerContext.java 0.00% 6 Missing ⚠️
...org/apache/amoro/maintainer/MaintainerMetrics.java 25.00% 6 Missing ⚠️
...pache/amoro/server/table/AbstractTableMetrics.java 0.00% 1 Missing ⚠️
...apache/amoro/server/table/DefaultTableRuntime.java 75.00% 1 Missing ⚠️
...n/src/main/java/org/apache/amoro/TableRuntime.java 0.00% 1 Missing ⚠️
...ache/amoro/maintainer/MaintainerOperationType.java 90.90% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #4045      +/-   ##
============================================
+ Coverage     29.93%   30.03%   +0.10%     
- Complexity     4229     4283      +54     
============================================
  Files           675      679       +4     
  Lines         53990    55072    +1082     
  Branches       6838     6981     +143     
============================================
+ Hits          16161    16541     +380     
- Misses        36636    37312     +676     
- Partials       1193     1219      +26     
Flag Coverage Δ
core 30.03% <57.95%> (+0.10%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@czy006 czy006 changed the title Draft: [AMORO-4034][Improvement]: MaintainerMetrics Unified interface and Report [AMORO-4034][Improvement]: MaintainerMetrics Unified interface and Report Jan 29, 2026
@czy006 czy006 requested a review from baiyangtx January 29, 2026 02:06
@czy006 czy006 force-pushed the issues/amoro-4034 branch 2 times, most recently from a4468f3 to 0cfa9a4 Compare January 29, 2026 08:56
Copy link
Copy Markdown
Contributor

@majin1102 majin1102 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this. Left some comments

@czy006 czy006 force-pushed the issues/amoro-4034 branch 4 times, most recently from d69deba to c63f243 Compare February 5, 2026 08:50
@czy006 czy006 requested a review from majin1102 February 5, 2026 13:25
@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 8, 2026

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@amoro.apache.org list. Thank you for your contributions.

@github-actions github-actions bot added the stale label Mar 8, 2026
@czy006 czy006 force-pushed the issues/amoro-4034 branch from 1853680 to 3eb6d57 Compare March 12, 2026 03:50
@github-actions github-actions bot added the type:docs Improvements or additions to documentation label Mar 12, 2026
@czy006
Copy link
Copy Markdown
Contributor Author

czy006 commented Mar 12, 2026

cc @baiyangtx @xxubai

@github-actions github-actions bot removed the stale label Mar 13, 2026
@czy006 czy006 force-pushed the issues/amoro-4034 branch from 0b7ec75 to d9da077 Compare April 9, 2026 14:36
@github-actions github-actions bot removed the type:docs Improvements or additions to documentation label Apr 9, 2026
@czy006 czy006 requested a review from Jzjsnow April 10, 2026 02:10
@czy006
Copy link
Copy Markdown
Contributor Author

czy006 commented Apr 10, 2026

@j1wonpark Can you help me review code ? This is an improvement to the monitoring abstraction of a multi-lake format.

}

private void expireSnapshots(long olderThan, int minCount, Set<String> exclude) {
// This method is kept for backward compatibility with potential subclasses
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This private method is no longer called after the refactor (2-param expireSnapshots now calls expireSnapshotsInternal directly). The "backward compatibility with subclasses" comment doesn't apply since it's private. Can be removed.

private static class OperationDurationGauge implements Gauge<Long> {
private volatile long value = 0L;
private volatile long startTime = 0L;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

startTime is written in recordStart() but never read — getValue() only returns value. Looks like a leftover from an earlier design. Can be removed along with recordStart().

@j1wonpark
Copy link
Copy Markdown
Contributor

j1wonpark commented Apr 10, 2026

autoCreateTags() logic is now inlined into IcebergTableMaintainer, but AutoCreateIcebergTagAction class and its test TestAutoCreateIcebergTagAction are not deleted in this PR. They become dead code after this change. Should be removed.

Also left a design suggestion about this in a separate comment below.

private void expireSnapshotsInternal(
long mustOlderThan, int minCount, MaintainerMetrics metrics) {
long startTime = System.currentTimeMillis();
Set<String> exclude = expireSnapshotNeedToExcludeFiles();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duration is recorded twice for the same operation — once by MaintainerOperationExecutor (table_maintainer_operation_duration_millis) and once inside this method (table_snapshots_expiration_duration_millis). The two values will differ slightly. Consider using a single source of truth for duration.

Same pattern applies to doExpireData and autoCreateTags.

@j1wonpark
Copy link
Copy Markdown
Contributor

Thanks for the contribution! I left a few inline comments on dead code and duration recording.

One design suggestion: the tag creation logic from AutoCreateIcebergTagAction is now inlined into IcebergTableMaintainer, which makes the class larger, leaves the original class as dead code, and couples tag creation logic with the maintainer. It might be cleaner to keep AutoCreateIcebergTagAction and extend it to support metrics, keeping the responsibilities decoupled.

@czy006 czy006 force-pushed the issues/amoro-4034 branch from 5bb09bd to b642e1d Compare April 13, 2026 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement]: MaintainerMetrics Unified interface and Report

4 participants