Pod in failed state doesn't clean up

[amy]

How to use it?

• kwok
• kwokctl --runtime=docker (default runtime)
• kwokctl --runtime=binary
• kwokctl --runtime=nerdctl
• kwokctl --runtime=kind

What happened?

Currently we're using kwok 0.5.1, wondering if upgrading would fix this issue.

So we have a pod that is in Failed state that should have terminated via the pod-remove-finalizer stage. Its underlying node is gone (which maybe affects things). But the pod still hangs around and kwok is unable to cleanup these pods.

What did you expect to happen?

The failed pod should be cleaned up after the test finishes. But the pod still exists with a kwok finalizer.

How can we reproduce it (as minimally and precisely as possible)?

unclear how to repro.

Anything else we need to know?

No response

Kwok version

Details

$ kwok --version
# paste output here

$ kwokctl --version
# paste output here

OS version

Details

```console # On Linux: $ cat /etc/os-release # paste output here $ uname -a # paste output here

On Darwin:

$ uname -a

paste output here

On Windows:

C:> wmic os get Caption, Version, BuildNumber, OSArchitecture

paste output here

</details>

[wzshiming]

Yes, the pod-remove-finalizer should be removed in later versions. If the node is deleted before the pod, kwok won't manage the orphaned pods.
You can also edit the stage directly to achieve your desired effect.

[amy]

Thanks so much for the quick response @wzshiming! Really appreciate it.

For upgrading, what is pod-remove-finalizer replaced with to clean up the completed pod? Specifically looking at these stages: https://github.com/kubernetes-sigs/kwok/tree/main/kustomize/stage/pod/general between completed and deleted, I'm not following where the finalizer is removed to allow the delete phase to match with the completed pod.

You can also edit the stage directly to achieve your desired effect.

We'll probably just upgrade. But just want to understand this option as well. What aspect of the stage can we edit to ensure the pod gets cleaned up?