Restart of Spark Operator may Result in Duplicated Submission

[kohn]

What happened?

• ✋ I have searched the open/closed issues and my issue is not listed.
  When I restart spark-operator, sometimes job fails with the error of "driver pod already exist". I checked the kubernetes event, and find the following event:
• T0: got the event of SparkApplicationSubmitted
• T1: old spark-operator exit
• T2: new spark-operator started
• T3: got the event of SparkApplicationAdded
• T4: got the event of SparkApplicationSubmissionFailed
• T5: got the event of SparkApplicationFailed

I guess this bug happens when spark-operator abruptly exist while spark-submit command is done but updateSparkApplicationStatus is not executed so that SparkApplication status is still "new"(""). Then the new spark-operator is up and try to re-submit again as the status is "new".

Reproduction Code

Keep submitting lots of jobs and restart the spark-operator

Expected behavior

jobs can be started

Actual behavior

some jobs fail with the error of "driver pod already exist"

Environment & Versions

• Kubernetes Version: 1.33
• Spark Operator Version: 2.3.0
• Apache Spark Version:

Additional context

No response

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

[tasier]

I’ve encountered this issue as well.

[rahul810050]

/assign

[Denyme24]

hey @kohn @ChenYi015, i would like to work on this issue .

make submission idempotent by checking for an existing driver pod before calling spark-submit, reuse it if it matches the SparkApplication, and only submit when no valid driver pod exists.

can i get assigned?

[rahul810050]

@Denyme24 I am currently working on this...

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.