Skip to content

[GOBBLIN-2268] Reconnect GobblinYarnAppLauncher to all YARN application states#4205

Open
thisisArjit wants to merge 1 commit into
apache:masterfrom
thisisArjit:gobblin-yarn-reconnect-all-states
Open

[GOBBLIN-2268] Reconnect GobblinYarnAppLauncher to all YARN application states#4205
thisisArjit wants to merge 1 commit into
apache:masterfrom
thisisArjit:gobblin-yarn-reconnect-all-states

Conversation

@thisisArjit

@thisisArjit thisisArjit commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

Description

  • Here are some details about my PR, including screenshots (if applicable):

RECONNECTABLE_APPLICATION_STATES only listed five of the eight YARN application states (NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING), so getReconnectableApplicationId() -- which queries
yarnClient.getApplications(APPLICATION_TYPES, RECONNECTABLE_APPLICATION_STATES) -- could never rediscover an application that had already moved into a terminal state (FINISHED, FAILED, KILLED). On driver restart such an application was treated as absent and a brand new one was launched instead of reconnecting to the existing one.

Use EnumSet.allOf(YarnApplicationState.class) so the launcher considers applications in every state when searching for one to reconnect to. This also keeps the behavior correct if new YARN application states are added in the future.

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

…in all states

RECONNECTABLE_APPLICATION_STATES only listed five of the eight YARN
application states (NEW, NEW_SAVING, SUBMITTED, ACCEPTED, RUNNING), so
getReconnectableApplicationId() -- which queries
yarnClient.getApplications(APPLICATION_TYPES, RECONNECTABLE_APPLICATION_STATES)
-- could never rediscover an application that had already moved into a
terminal state (FINISHED, FAILED, KILLED). On driver restart such an
application was treated as absent and a brand new one was launched
instead of reconnecting to the existing one.

Use EnumSet.allOf(YarnApplicationState.class) so the launcher considers
applications in every state when searching for one to reconnect to. This
also keeps the behavior correct if new YARN application states are added
in the future.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@thisisArjit thisisArjit marked this pull request as ready for review July 15, 2026 06:19
@thisisArjit thisisArjit changed the title [GOBBLIN-XXXX] Reconnect GobblinYarnAppLauncher to YARN applications … [GOBBLIN-XXXX] Reconnect GobblinYarnAppLauncher to all YARN application states Jul 15, 2026
@thisisArjit thisisArjit changed the title [GOBBLIN-XXXX] Reconnect GobblinYarnAppLauncher to all YARN application states [GOBBLIN-2268] Reconnect GobblinYarnAppLauncher to all YARN application states Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant