App health stuck at "Progressing" forever when pod has `restartPolicy: Never`

[lindhe]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

When an Application deploys a pod that has restartPolicy: Never, the Application gets stuck with health "Progressing" seemingly forever.

To Reproduce

Apply this manifest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  name: debug
  namespace: argocd
spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: default
  source:
    path: manifests
    repoURL: https://github.com/lindhe/debug-argocd-15317.git

That app installs just this seemingly innocuous pod:

apiVersion: v1
kind: Pod
metadata:
  name: foo
spec:
  containers:
    - image: busybox
      command: ['sh', '-c', 'sleep 3600']
      name: bar
  restartPolicy: Never

Expected behavior

The application health should soon become "Healthy".

Screenshots

Version

argocd: v2.7.9+0ee33e5
  BuildDate: 2023-07-24T18:26:12Z
  GitCommit: 0ee33e52dd1f1bb944488584fc6f854b929f1180
  GitTreeState: clean
  GoVersion: go1.19.11
  Compiler: gc
  Platform: linux/amd64
WARN[0000] Failed to invoke grpc call. Use flag --grpc-web in grpc calls. To avoid this warning message, use flag --grpc-web.
argocd-server: v2.8.0+804d4b8

Logs

No relevant logs that I've found.

Other information

Similar to #5620 and some other issues found by searching for "health progressing".

I'm running an RKE2 cluster with version v1.26.6+rke2r1 and Cilium 1.13.2 installed.

[blakepettersson]

This seems to be by design, or rather there is nothing currently better than what there is (see the relevant snippet in gitops engine).

[lindhe]

Ah, that's interesting. Thanks for finding it!

It sounds to me like ignoring all pods that sets restartPolicy: Never or restartPolicy: OnFailure is a very blunt tool to achieve those ends. Perhaps we can find a better way of doing things?

After reading the documentation, it is my understanding that all resource hooks have the argocd.argoproj.io/hook annotation. Perhaps we could therefore instead look for pods that fulfills these requirements:

• has restartPolicy: Never or restartPolicy: OnFailure.
• has the argocd.argoproj.io/hook annotation or is owned by an object with that annotation.

Do you think it sounds reasonable to look for an alternative solution like this, in order to avoid these unexpected behaviors?

[blakepettersson]

Perhaps we could therefore instead look for pods that fulfills these requirements:

From my perspective that does sound reasonable. I suspect that there might be a caveat or two to this which would still make this a no-go. @jessesuen do you have any input regarding this?

[blakepettersson]

@lindhe perhaps the way to move this forward is to come in one of the Argo CD Contributors meetings and make a proposal?

[lindhe]

Sure, I'd love to. Where do I find the schedule?

[blakepettersson]

You can see the agenda and Zoom link here. TLDR every Thursday at 17.15 CET

[lindhe]

Great. I don't have time this week, but expect me soon! :)

[nweisenauer-sap]

Hey there, any updates on this one? @lindhe

[lindhe]

Sorry, I dropped the ball on this one. 😕 Q4 last year was hectic for me. Still plan to push for it on the contributors meeting, but if someone has time to join and discuss it before I have time, go for it! 👍

[jpuyy]

I also met this problem for flink task manager, it may affect the flink autoscale.

flink operator release-1.7.0