[Bug] Doris 4.1.0 leaks two Ranger PolicyRefresher threads after every metadata checkpoint

[RoggeDeior]

Search before asking

• I had searched in the issues and found no similar issues.

Version

DORIS_BUILD_VERSION=doris-4.1.0-rc03
DORIS_BUILD_HASH=5960d4cea0e6d2a08f62b45a07a662395d2bbcfc
Java=17.0.2
Deployment=Kubernetes
Mode=Cloud / compute-storage decoupled
Authorization=Apache Ranger
Ranger version=2.7.0
access_controller_type=ranger-doris

The build hash is the same commit used by the Doris 4.1.0 release tag.

What's Wrong?

I searched existing issues and found:

• [Bug] doris create more and more ranger policy refresher  #45641
• [fix](ranger) make RangerDorisAccessController as singleton to avoid more and more ranger policy refresher #45645

PR #45645 previously made RangerDorisAccessController a singleton. However, this fix is no longer effective in Doris 4.1.0 because the new RangerDorisAccessControllerFactory directly creates a new controller for every checkpoint Env.

Every successful metadata checkpoint creates two new Ranger PolicyRefresher threads. These threads are not stopped when the temporary checkpoint Env is destroyed.

The number of threads therefore increases continuously:

PolicyRefresher count
= 1 normal refresher created by main
+ checkpoint count × 2

In our test environment:

Journal-triggered checkpoints:       3
Cloud stale-image checkpoints:     135
Total checkpoints:                 138

PolicyRefresher created by main:      1
PolicyRefresher created by
leaderCheckpointer:                 276

Total PolicyRefresher threads:      277

The numbers match exactly:

1 + 138 × 2 = 277

Thread creator distribution from the FE logs:

Count  Thread
-----  ------------------
276    leaderCheckpointer
1      main

The creation rate also matches the default cloud checkpoint interval. With:

cloud_checkpoint_image_stale_threshold_seconds = 3600

one cloud checkpoint is triggered every hour, and two new PolicyRefresher threads are leaked per checkpoint, resulting in approximately 48 additional refresher threads per day.

Evidence from FE Logs

A checkpoint directly creates a new Ranger plugin and PolicyRefresher:

RuntimeLogger 2026-07-07 11:10:57,122 INFO
(leaderCheckpointer|143)
[Checkpoint.doCheckpoint():122]
Trigger checkpoint since last checkpoint journal id: 5117761
is less than current finalized journal id: 5162245

RuntimeLogger 2026-07-07 11:10:57,140 INFO
(leaderCheckpointer|143)
[Checkpoint.doCheckpoint():155]
begin to generate new image: image.5162245

RuntimeLogger 2026-07-07 11:10:57,151 INFO
(leaderCheckpointer|143)
[AccessControllerManager.loadAccessControllerPlugins():102]
Found Authentication Plugin Factories: ranger-doris from class path.

RuntimeLogger 2026-07-07 11:10:57,159 INFO
(leaderCheckpointer|143)
[RangerBasePlugin.init():308]
Created PolicyRefresher Thread(PolicyRefresher(serviceName=doris)-377)

Every checkpoint leaves two additional PolicyRefresher threads running permanently.

In our production environment, the FE accumulated:

1321 PolicyRefresher threads

At the default 30-second Ranger polling interval, this generated approximately:

1321 refreshers × 2 Ranger endpoints / 30 seconds
≈ 88 requests per second

The affected endpoints included:

/service/plugins/policies/download/doris
/service/roles/download/doris

This eventually caused Ranger Admin to accumulate millions of asynchronous plugin-info update tasks:

RangerTransactionService:
scheduled = 20,751,852
pending   = 5,334,642

Ranger Admin then experienced:

Old generation usage: approximately 96%
Full GC count: 5197
Full GC time: approximately 18 hours
java.lang.OutOfMemoryError: GC overhead limit exceeded

Tomcat Acceptor/Poller threads eventually disappeared, while port 6080 remained in LISTEN state with a full TCP accept backlog, making Ranger Admin appear running but inaccessible.

What You Expected?

Only one Ranger PolicyRefresher should exist per Doris FE process.

Creating or destroying a temporary checkpoint Env must not leave any long-running Ranger background threads.

How to Reproduce?

1. Deploy Doris 4.1.0 in Cloud Mode.

2. Configure Ranger authorization:

access_controller_type = ranger-doris

3. Keep the default configuration:

cloud_checkpoint_image_stale_threshold_seconds = 3600

4. Start FE and wait for several cloud metadata checkpoints.

5. Count Ranger PolicyRefresher threads:

jcmd <FE_PID> Thread.print \
  | grep -c '^"PolicyRefresher(serviceName=doris)'

6. Count checkpoint events:

grep -R -h \
  'Trigger checkpoint since' \
  /opt/apache-doris/fe/log | wc -l

grep -R -h \
  'Trigger checkpoint in cloud mode because latest image is expired' \
  /opt/apache-doris/fe/log | wc -l

7. Count PolicyRefresher creation records:

grep -R -h \
  'Created PolicyRefresher Thread(PolicyRefresher(serviceName=doris)' \
  /opt/apache-doris/fe/log | wc -l

The PolicyRefresher count grows by two after every successful checkpoint.

Root Cause Analysis

The observed call chain is:

Checkpoint.doCheckpoint()
  -> Env.getCurrentEnv()
  -> EnvFactory.createEnv(true)
  -> new checkpoint Env
  -> new AccessControllerManager
  -> RangerDorisAccessControllerFactory.createAccessController()
  -> new RangerDorisAccessController("doris")
  -> RangerBasePlugin.init()
  -> new PolicyRefresher

Checkpoint.doCheckpoint() obtains and destroys a checkpoint Env twice during one successful checkpoint.

The relevant Env.getCurrentEnv() bytecode is equivalent to:

if (isCheckpointThread()) {
    if (CHECKPOINT == null) {
        CHECKPOINT = EnvFactory.getInstance().createEnv(true);
    }
    return CHECKPOINT;
}

However, Env.destroyCheckpoint() only clears the static reference:

public static void destroyCheckpoint() {
    if (CHECKPOINT != null) {
        CHECKPOINT = null;
    }
}

It does not close AccessControllerManager, clean up the Ranger plugin, or stop PolicyRefresher.

The Doris 4.1.0 factory directly creates a new controller:

https://github.com/apache/doris/blob/4.1.0/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/RangerDorisAccessControllerFactory.java

@Override
public RangerDorisAccessController createAccessController(
        Map<String, String> prop) {
    return new RangerDorisAccessController("doris");
}

RangerDorisAccessController in Doris 4.1.0 also no longer contains the singleton getInstance() implementation introduced by PR #45645.

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_84bc83b0a52a
This content is generated by AI for reference only.

Initial maintainer-facing analysis:

I would treat this as a valid FE lifecycle bug, not only a Ranger Admin-side capacity problem. I checked the reported release commit 5960d4cea0e6d2a08f62b45a07a662395d2bbcfc / 4.1.0-rc03, and the code path matches the reported "+2 PolicyRefresher threads per successful checkpoint" pattern.

Code evidence:

• Checkpoint.doCheckpoint() creates a checkpoint Env for replay/save, destroys it after saving the image, then creates another checkpoint Env to load and verify the generated image. In the 4.1.0 tag this is the Env.getCurrentEnv() path around Checkpoint.java lines 156 and 182, with Env.destroyCheckpoint() around lines 176 and 199.
• Env.getCurrentEnv() creates the checkpoint environment via EnvFactory.getInstance().createEnv(true) when the current thread is the checkpoint thread.
• The Env constructor always creates a new AccessControllerManager(auth).
• AccessControllerManager loads the configured default access controller during construction. With access_controller_type=ranger-doris, RangerDorisAccessControllerFactory.createAccessController() returns new RangerDorisAccessController("doris").
• RangerDorisAccessController constructs a RangerDorisPlugin, and RangerDorisPlugin calls RangerBasePlugin.init() in its constructor. That is the path that starts the Ranger policy refresher.
• Env.destroyCheckpoint() only clears the static CHECKPOINT reference. In this branch, AccessControllerManager / CatalogAccessController do not expose a close lifecycle, and the Ranger controller path does not call RangerBasePlugin.cleanup().

So one successful checkpoint can create two temporary Ranger Doris plugins, and neither is explicitly cleaned up. That makes the reported count formula:

1 main FE plugin + 2 * successful checkpoint count

consistent with the current code.

I also checked refreshed upstream/branch-4.1 and upstream/master; both still show RangerDorisAccessControllerFactory returning new RangerDorisAccessController("doris"), and the checkpoint cleanup shape is still the same. The older public singleton fixes referenced by the issue do not appear to be ancestors of 4.1.0, branch-4.1, or master in the refreshed local repository state I inspected.

Suggested next steps:

1. Fix this on master and branch-4.1, then backport to affected 4.1.x release lines.
2. Prefer preventing checkpoint Env construction from creating independent long-lived Ranger Doris plugins. Restoring a process-wide singleton for the internal Doris Ranger controller/plugin is the narrowest direction and matches the previous fix intent.
3. If maintainers prefer a general lifecycle fix instead, add an explicit close path from Env.destroyCheckpoint() down through AccessControllerManager / access controllers and call Ranger plugin cleanup for checkpoint-owned plugins. That path needs care because the serving FE plugin must not be cleaned by temporary checkpoint Env teardown.
4. Add a regression test around checkpoint Env creation/destruction with access_controller_type=ranger-doris. The test should assert that one successful checkpoint does not leave two additional live Ranger plugin refresher instances, or that checkpoint-owned Ranger plugins are explicitly cleaned.
5. Audit similar access-controller factories for background-thread ownership, especially any Ranger-backed controller that can be instantiated while replaying or verifying checkpoint metadata.

Operational mitigation until a fix is available:

• Restarting the affected FE clears the accumulated leaked refresher threads in that FE process.
• Increasing cloud_checkpoint_image_stale_threshold_seconds can reduce the cloud stale-image checkpoint leak rate, but it only slows the leak and changes image freshness behavior.
• Increasing the Ranger policy polling interval can reduce Ranger Admin request pressure, but it does not fix the leaked FE threads.

Additional information that would be useful, but is not required to accept the bug:

• A thread dump before and after two consecutive successful checkpoints from the same FE process.
• FE logs covering several consecutive checkpoints, including both checkpoint trigger lines and Created PolicyRefresher Thread(...) lines.
• The effective Ranger polling interval and Ranger Admin endpoint count, to quantify operational impact.