KCP does not remove the etcd member for a machine that failed to join the control plane

[dlipovetsky]

What steps did you take and what happened?

1. Create a cluster with a 3-node control plane.
2. Create a machine image that is misconfigured, so that the kubelet cannot start.
3. Update KubeadmControlPlane to use this image.
4. When kubeadm join --control-plane runs, it adds an etcd member as a learner, tries to promote it, and eventually fails, because etcd is not running, since kubelet is not running. The etcd member remains unstarted.
5. At some point, KCP remediation will delete the machine that has failed to join the control plane. However, KCP will not remove the etcd member for this machine, because the machine has no NodeRef, as it never joined the cluster.
6. KCP will create a new machine, and when kubeadm join --control-plane runs, it will try to add a member as a learner, and that will fail, because no etcd members can be added, while there is one unstarted etcd member.

What did you expect to happen?

KCP should remove the unstarted etcd member that was added for the machine that failed to join the cluster.

Cluster API version

v1.10.7

Kubernetes version

v1.34.2

Anything else you would like to add?

This issue came out of a discussion in kubernetes/kubeadm#3269 (comment)

Label(s) to be applied

/kind bug
/area provider/control-plane-kubeadm

[sbueringer]

Thx for the report. Do we know if there is a safe way to determine the correct etcd member to delete during remediation?

[dlipovetsky]

Thx for the report. Do we know if there is a safe way to determine the correct etcd member to delete during remediation?

KCP creates control plane replicas sequentially. When KCP creates a replica (i.e. creates a Machine), then previously created replicas have already joined the control plane, and therefore the etcd members are started (even members are split from their peers at some later point). At most one previously created replica failed to start etcd. Therefore, there is at most one unstarted learner member. It is safe for KCP to delete this unstarted learner member.

(It should delete it before creating the Machine resource. If it deletes it after creating the Machine resource, but is interrupted, then at the next reconcile, it will be unable to delete it, because it has already created the Machine resource)

[fabriziopandini]

might be there are other places where we can do this, e.g. in reconcilePreTerminateHook (around when we call remove etcd member for machines)

[fabriziopandini]

/help

[k8s-ci-robot]

@fabriziopandini:
This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

Details

In response to this:

/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.