[pull] master from kubernetes:master#1924
Open
pull[bot] wants to merge 8318 commits into
Open
Conversation
kops-controller and nodeup bug fixes
Signed-off-by: Jefftree <jeffrey.ying86@live.com>
nodeup: fix protokube skip on hybrid-bootstrap workers
channels: stop distributing the standalone binary
Renders the kops-channels pod at cloudup so the image string is routed through AssetBuilder.RemapImage, picking up containerRegistry / containerProxy overrides and digest pinning. Mirrors the etcd-manager pattern: build the pod, remap the image, store the manifest under manifests/channels/ in the state store. The bootstrap channel URL is constructed via vfs path joining so it matches toolbox_enroll's identity comparison exactly. Subsequent commits wire nodeup to copy this manifest in place of generating its own. Rejects file:// in cluster.Spec.Addons[].manifest at validation time: the static pod fetches channels via VFS, and a file:// URL would resolve inside the container's mount namespace rather than on the host. The state store or another VFS backend is the supported path.
Adds NodeupConfig.ChannelsManifest pointing at the state-store path where the cloudup builder writes the kops-channels manifest. Populated on control-plane instance groups. Also removes NodeupConfig.Channels (and the populating loop / builder state): it was only ever read by protokube as the list to apply on a 60s loop, but the kops-channels static pod's args now carry that list — baked at cloudup, not derived from this field. toolbox_enroll switches to deriving the bootstrap URL from configStore.base in the next commit.
ChannelsBuilder now reads the manifest from the state-store path supplied by NodeupConfig.ChannelsManifest and writes it to /etc/kubernetes/manifests/. The pod itself, including the remapped image, is produced at cloudup. SELinux decoration is still applied at nodeup so per-instance-group overrides keep working, and a table-driven test pins both the ON and OFF branches via a structured Pod field check. The user and kubeconfig tasks remain host-side. toolbox enroll also gains handling for the new ChannelsManifest field in a single block: derive the bootstrap URL from configStore.base, download the addons tree and the pod manifest, then rewrite the bootstrap URL inside the manifest to file://<local addons> and add the matching hostPath mount. The s3:// precondition is explicit, and the rewriter logs a warning if no arg matched the bootstrap URL so cloudup-manifest drift is visible. Without this the enrolled control-plane node would either fetch the bootstrap channel from the remote VFS URL it has no credentials for, or hit ENOENT because the channels container has no view of the host addons tree. The pre-existing Channels[] loop is gone — its only effect was mutating nodeupConfig.Channels[i] (now removed from the API), and the addons remapTree call is folded into the new block. Drops the stale KUBECONFIG entry from protokube's sysconfig env file: protokube no longer talks to the apiserver, so the kubeconfig is the kops-channels container's concern alone.
Two golden fixtures lock the load-bearing image-remap path: - minimal: baseline pod, default registry image - container_registry: spec.assets.containerRegistry must rewrite the image to my-mirror.example.com/kops-channels:... — the regression that motivated this refactor
Adds aws_s3_object_manifests-channels-kops-channels_content to the AWS, Hetzner, and Scaleway expected-file lists, and reruns hack/update-expected to refresh the nodeupconfig, kubernetes.tf, and per-cluster launch-template snapshots. NodeupConfig.ChannelsManifest is added and NodeupConfig.Channels is dropped, so every nodeupconfig snapshot picks up both changes. Signed-off-by: Ciprian Hacman <ciprian@hakman.dev>
kubetest2-kops: fix GCE DNS truncation
channels: build static pod manifest in cloudup
`kops get assets` runs with checkExisting disabled, so DNSZone.Find is skipped and ZoneID is never populated from a DNS-name dnsZone. IAMRolePolicy.ShouldCreate then aborts the run with "DNS ZoneID not set" before assets.Copy is reached. Fall back to DNSName when ZoneID is empty. The rendered policy is discarded in dry-run modes; real applies always populate ZoneID before this path runs. Signed-off-by: Ciprian Hacman <ciprian@hakman.dev>
Fix kops get assets when spec.dnsZone is a DNS name
kops-channels, kops-controller, and etcd-manager run as non-root and can't read /sys/devices/virtual/dmi/id/product_uuid (mode 0400), so util/pkg/vfs/s3context.go's isRunningOnEC2 returns false, skips IMDS, defaults to us-east-1, and pays for a cross-region GetBucketLocation / HeadBucket round-trip on every cold start before learning the bucket's real region. Derive AWS_REGION from the cluster spec's subnet zones (kops validation guarantees they all share a region) so the generated pod manifest carries the right value. Stubs that don't ship subnets (the nodeup-side ChannelsBuilder) fall back to AWS_REGION from the calling process env, matching the existing S3_REGION pattern. Signed-off-by: Ciprian Hacman <ciprian@hakman.dev>
kops-channels runs as UID 10013 with no HOME, so client-go's cached discovery falls back to /.kube/cache and logs ~22 "mkdir /.kube: permission denied" lines on every pod start before silently giving up. Point HOME at /tmp; chainguard/static (the ko base image) creates /tmp mode 1777 via wolfi-baselayout, so the non-root uid can write there via the container's writable overlay. No emptyDir needed. Signed-off-by: Ciprian Hacman <ciprian@hakman.dev>
The aws-cloud-controller-manager and kops-controller manifests still
list node-role.kubernetes.io/master alongside node-role.kubernetes.io/
control-plane in nodeSelectorTerms. The apiserver emits a deprecation
warning on every channel apply ("node-role.kubernetes.io/master is
use \"node-role.kubernetes.io/control-plane\" instead"). kops itself
hasn't applied the master label to control-plane nodes for years, so
the fallback term selects nothing and is dead weight.
Drop the master matchExpressions block. Keep the corresponding
toleration as harmless backward-compat for any straggler node that
still carries the legacy taint.
Signed-off-by: Ciprian Hacman <ciprian@hakman.dev>
Channels' health check warned "expected status.conditions to be list, got <nil>" whenever a CRD set status.conditions to null instead of omitting the key. The two amazon-vpc-routed-eni CRDs trigger this on every channel apply. Fold the nil case into the existing "status conditions not found" info path; both are semantically "object has no conditions to report". Signed-off-by: Ciprian Hacman <ciprian@hakman.dev>
channels: fix region detection and discovery cache permission noise
Validate instance group names in kops-controller
Upgrade Go to 1.26.3
Signed-off-by: Ciprian Hacman <ciprian@hakman.dev>
Update dependencies
Replace the hardcoded module lists in the `gomod` make target and verify-gomod.sh with dynamic discovery, so no module is left untidied or unverified. Hidden directories are excluded.
chore: update syntax of CLAUDE.md
azure: Bump azure-cloud-controller-manager to v1.36.2
The built-in Owner role includes Microsoft.Authorization/roleAssignments/write, a privilege-escalation path reachable by any pod via the instance metadata endpoint. No cluster component uses Microsoft.Authorization; Contributor covers everything the control plane needs. Existing clusters keep the old Owner role assignment until it is removed manually or the cluster is deleted; the terraform target prunes it on apply.
azure: Grant control-plane VMSS Contributor instead of Owner
The hostname topologySpreadConstraint uses whenUnsatisfiable: DoNotSchedule, but CoreDNS autoscales via the cluster-proportional-autoscaler. On clusters with a single schedulable worker (e.g. one control-plane + one worker), the tainted control-plane node was counted as an empty topology domain, so the second CoreDNS replica could never satisfy maxSkew and stayed Pending. Setting nodeTaintsPolicy: Honor excludes nodes with untolerated taints (control-plane, cordoned/draining) from the skew calculation, keeping the hard spread across workers while allowing the replicas to co-locate when there is only one eligible node.
coredns: Honor node taints in hostname topologySpreadConstraint
ClusterLoader2's managed Prometheus client (clusterloader2/pkg/prometheus/clients/gcp_managed.go) reads the GCP project from the PROJECT env var to build the Cloud Monitoring query URL. The kubetest2-kops deployer only exported GCP_PROJECT, so for scalability jobs that enable the "Quotas total usage" measurement (via preset-e2e-scalability-common's CL2_ENABLE_QUOTAS_USAGE_MEASUREMENT) the query URL had an empty project and Cloud Monitoring rejected it with HTTP 400 INVALID_ARGUMENT, failing the load test. Also export PROJECT alongside GCP_PROJECT so the measurement can query the correct project. This is set from d.GCPProject, which is populated even when the project is acquired from boskos.
hetzner: improve Cluster Autoscaler integration The Cluster Autoscaler Hetzner provider keys node group membership, initial target size and Nodes() entirely off the hcloud/node-group server label, which kOps never set. As a result kOps-created servers were invisible to the autoscaler. - Label Node-role servers with hcloud/node-group at creation. Servers from earlier kOps versions get the label when they are replaced. - Treat minSize as a floor: servers beyond Count are no longer marked needs-update, so autoscaler scale-ups survive kops update cluster. Shrink with 'kops delete instance' or autoscaler scale-down. Signed-off-by: Ciprian Hacman <ciprian@hakman.dev>
hetzner: fix Cluster Autoscaler node group membership
The "Services should implement NodePort and HealthCheckNodePort correctly when ExternalTrafficPolicy changes" test fails on calico on Azure for the same reason it does on GCE: the underlay cannot route the pod CIDR, so calico must encapsulate inter-node pod traffic (VXLAN vxlan.calico on Azure, IPIP tunl0 on GCE). On the node-local ExternalTrafficPolicy=Local NodePort short-circuit path the masquerade then rewrites the client source IP to the node's tunnel address (a pod-CIDR IP) instead of preserving it, so the test fails on every run once the cluster comes up. Calico preserves the source IP only on AWS, where kOps disables the source/dest check and routes pod traffic natively. Extend the calico skip from gce to also cover azure.
Expose GCP project as PROJECT env for ClusterLoader2 on GCE
Signed-off-by: Ciprian Hacman <ciprian@hakman.dev>
tests/e2e: skip implement-NodePort ETP=Local test on calico+Azure
Update cluster-autoscaler to v1.36.0
The nodes instance group in this scenario spans two zones, so kOps creates two zonal InstanceGroupManagers, but only the first zone's MIG is registered with cluster-autoscaler. The expected output captures this current behavior.
…ter-autoscaler For a GCE instance group spanning multiple zones, kOps creates one zonal InstanceGroupManager per zone, but only registered the first zone's MIG with cluster-autoscaler, carrying the full min/max of the instance group. The MIGs in the remaining zones were invisible to the autoscaler and were never scaled, so scale-ups could get stuck in a single zone (for example when that zone has no spot capacity). Register each zonal MIG as its own cluster-autoscaler node group, splitting the instance group's min/max sizes across zones with the same algorithm used for the MIG target sizes. Single-zone instance groups render exactly as before.
gce: register all zonal MIGs of a multi-zone instance group with cluster-autoscaler
Update Go to 1.26.5 and bump golang.org/x modules
pkg/assets: keep container registry clients out of runtime binaries
clusterapi: fix build by making gen.go a library package
Support running custom etcd versions by setting version and image
Default to etcd 3.7.0 for Kubernetes 1.37+
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot]
Can you help keep this open source service alive? 💖 Please sponsor : )