After a node becomes unready and all it's pod are marked as unready, KWOK fails to update pods to ready

[serathius]

How to use it?

• kwok
• kwokctl --runtime=docker (default runtime)
• kwokctl --runtime=binary
• kwokctl --runtime=nerdctl
• kwokctl --runtime=kind

What happened?

Noticed that with large number of nodes (4k), KWOK sometimes fails to update node lease resulting to node become unready. This is resolved after automatically, but there is a side effect on pods.

When node becomes unready all it's pod are marked unready too. KWOK fails to update the pods to be ready back again, resulting in some pods to be stuck in unready state forever.

Example:

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2025-10-09T08:21:21Z"
  generateName: nginx-
  generation: 1
  labels:
    app: nginx
    apps.kubernetes.io/pod-index: "3"
    controller-revision-hash: nginx-d6df65d5b
    statefulset.kubernetes.io/pod-name: nginx-3
  name: nginx-3
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: nginx
    uid: 14bac6df-c3ed-43db-b636-4a8587aa75e3
  resourceVersion: "496007"
  uid: ea786112-54bf-4761-a1da-dce13ab4c265
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: type
            operator: In
            values:
            - kwok
  containers:
  - image: registry.k8s.io/nginx-slim:0.21
    imagePullPolicy: IfNotPresent
    name: nginx
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-jr9h9
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: nginx--3
  nodeName: kwok-node-1
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoSchedule
    key: kwok.x-k8s.io/node
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-jr9h9
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2025-10-09T08:22:16Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2025-10-09T08:27:40Z"
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2025-10-09T08:22:16Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2025-10-09T08:22:14Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: registry.k8s.io/nginx-slim:0.21
    imageID: ""
    lastState: {}
    name: nginx
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: "2025-10-09T08:22:16Z"
  hostIP: 10.0.0.1
  phase: Running
  podIP: 10.0.16.123
  podIPs:
  - ip: 10.0.16.123
  qosClass: BestEffort
  startTime: "2025-10-09T08:22:16Z"

What did you expect to happen?

KWOK should ensure that all pods it's responsible for have ready condition true.

How can we reproduce it (as minimally and precisely as possible)?

1. Create cluster with KWOK (in my case KIND cluster with KWOK outside of cluster). Ensure all nodes are ready:

   $ kubectl get nodes kwok-node-1
   NAME               STATUS   ROLES   AGE   VERSION
   kind-control-plane Ready      control-plane   45m   v1.34.1-dirty
   kwok-node-1        Ready    agent   42m   fake
   

2. Schedule pods (in my case via statefulsefulset) and wait for them to become ready. Can be confirmed by running kubectl get statefulset. Like:

   $ kubectl get statefulset nginx
   NAME        READY   AGE
   nginx       4/4     31m
   

3. Stop KWOK for couple for seconds enough for nodes, start it again and notice node flip between NotReady and Ready.

   $ kubectl get nodes 
   NAME                 STATUS     ROLES           AGE   VERSION
   kind-control-plane   Ready      control-plane   45m   v1.34.1-dirty
   kwok-node-1          NotReady   agent           44m   fake
   

4. After Node becomes ready notice that pods never become ready.

   $ kubectl get statefulset nginx
   NAME        READY   AGE
   nginx       0/4     36m
   

Anything else we need to know?

No response

Kwok version

Details

$ kwok --version
kwok version v0.7.0 go1.24.1 (linux/amd64)

OS version

N/A

[serathius]

I think the problem might be related to KWOK not trying to imitate proper Kubelet behavior, just doing minimal work to mark nodes as ready. This makes it not great to scale test controllers that might depend on shape and number of pod reconcilation steps. I would suggest to consider looking into bringing KWOK closer to abstracing Kubelet pod state machine.

[wzshiming]

You need to modify the pod-ready selector in the stage-fast.yaml file.

It needs to be deleted.

kwok/kustomize/stage/pod/fast/pod-ready.yaml

Lines 13 to 14 in 8e1b459

	- key: '.status.podIP'
	operator: 'DoesNotExist'

[wzshiming]

If you need a full simulation of the kubelet's behavior, you're probably looking for the Kubemark.

[serathius]

I think there is some marketing and scope problems:

[wzshiming]

Make it more like a real pod. This feature is on roadmap #1406, but it hasn't been developed yet due to low demand. The main requirements are scheduling, autoscaling, customizable behavior simulation, and testing.

[wzshiming]

https://kwok.sigs.k8s.io/

The difference is described in the website

[serathius]

Don't know what you mean by Make it more like a real pod. A pod without readiness probe should always be ready if it's running. Or am I missing something? That sounds like a basic Kubelet behavior.

While I appreciate the customizability of KWOK to test different scenarios, I don't understand why the default behavior doesn't follow simple Kubelet logic of: Pod should be marked as running and ready. That would mean that kwok doesn't fulfill it's goal of simulating behavior of a node.

[wzshiming]

Kwok allows users to modify the pod status themselves to verify the various responses of the controllers. If it is forced to stay in the running state, however, this feature will be lost.

As I mentioned #1464 (comment)
You only need to delete this, which will force it to stay in the running state.

And the original definition of a real pod for this project was simply the ability to be scheduled and run.

[wzshiming]

I think #1406 is doing exactly what you described regarding the Kubelet's behavior. There are no users or feedback, so I won't continue. If there were, I think I'd be working on it.

[serathius]

I think bringing the project closer (maybe not exact) to what Kubelet is doing will help with adoption for other components.

Similar to how SIG scheduling created its own dedicated benchmarks, I would like to start decomposing scalability tests by individual components as proposed in kubernetes/kubernetes#134375. Testing things like kube-controller-manager for a maximum number of statefulsets supported, so we can finally start filling in TBD cells in https://github.com/kubernetes/community/blob/master/sig-scalability/configs-and-limits/thresholds.md#kubernetes-thresholds.

I think decomposition of scalability by components and using tools like kwok as fakes will be crucial in defining and maintaining the scalability envelope. Not everyone has access to 5k VMs to check if their component scales, using fakes should make cost to entry much lower.

[wzshiming]

That sounds great, and I'd be happy to help. everything in kwok is configurable, and has even helped KubeVirt fakes VM instances before.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten