Skip to content

[CELEBORN-2349] Support worker supplied tags and TagsManager hardening#3719

Open
s0nskar wants to merge 3 commits into
apache:mainfrom
s0nskar:tags_improvements
Open

[CELEBORN-2349] Support worker supplied tags and TagsManager hardening#3719
s0nskar wants to merge 3 commits into
apache:mainfrom
s0nskar:tags_improvements

Conversation

@s0nskar
Copy link
Copy Markdown
Contributor

@s0nskar s0nskar commented Jun 5, 2026

What changes were proposed in this pull request?

This PR lets a worker advertise tags at startup and makes tags part of WorkerInfo, so they will use existing Raft replication and worker-lifecycle cleanup. This also gel well with current implementation, allowing tags from config service to be merged with worker supplied tags.

Backward Compatible:

  • The behaviour remains unchanged when no worker supplied tags are configured.
  • A master-side config celeborn.tags.worker.registration.enabled (default: true), will allow to ignore worker-supplied tags.
  • If worker is running on older version, worker supplied tags will be empty.

Why are the changes needed?

TagsManager today only works off the dynamic-config store. There is no way for a worker to declare its own tags. This makes this feature a little hard to use.

After this PR a worker can supply its own tags and manage its own tags lifecycle and still allow tags overriding via config service. This will make worker tags feature a lot easier to use.

Does this PR resolve a correctness bug?

  • Yes

Does this PR introduce any user-facing change?

  • Yes

How was this patch tested?

  • Added Unit Tests.
  • Working on end-to-end testing in local setup.

@s0nskar s0nskar changed the title [CELEBORN-2349] Support worker supplied tags during startup and TagsManager hardening [CELEBORN-2349] Support worker supplied tags and TagsManager hardening Jun 5, 2026
@s0nskar
Copy link
Copy Markdown
Contributor Author

s0nskar commented Jun 5, 2026

@SteNicholas can you please trigger Copilot magic on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant