diff --git a/docs/en/apis/providers/bare-metal/baremetalcluster.mdx b/docs/en/apis/providers/bare-metal/baremetalcluster.mdx new file mode 100644 index 00000000..26c97314 --- /dev/null +++ b/docs/en/apis/providers/bare-metal/baremetalcluster.mdx @@ -0,0 +1,8 @@ +--- +weight: 10 +--- + +# BaremetalCluster [infrastructure.cluster.x-k8s.io/v1beta1] + +{/* cspell:disable-next-line */} + diff --git a/docs/en/apis/providers/bare-metal/baremetalmachine.mdx b/docs/en/apis/providers/bare-metal/baremetalmachine.mdx new file mode 100644 index 00000000..91890f4e --- /dev/null +++ b/docs/en/apis/providers/bare-metal/baremetalmachine.mdx @@ -0,0 +1,8 @@ +--- +weight: 20 +--- + +# BaremetalMachine [infrastructure.cluster.x-k8s.io/v1beta1] + +{/* cspell:disable-next-line */} + diff --git a/docs/en/apis/providers/bare-metal/baremetalmachinetemplate.mdx b/docs/en/apis/providers/bare-metal/baremetalmachinetemplate.mdx new file mode 100644 index 00000000..7cc878de --- /dev/null +++ b/docs/en/apis/providers/bare-metal/baremetalmachinetemplate.mdx @@ -0,0 +1,8 @@ +--- +weight: 30 +--- + +# BaremetalMachineTemplate [infrastructure.cluster.x-k8s.io/v1beta1] + +{/* cspell:disable-next-line */} + diff --git a/docs/en/apis/providers/bare-metal/index.mdx b/docs/en/apis/providers/bare-metal/index.mdx index 55818264..0f07efb5 100644 --- a/docs/en/apis/providers/bare-metal/index.mdx +++ b/docs/en/apis/providers/bare-metal/index.mdx @@ -6,14 +6,37 @@ weight: 40 -## Status +The Bare Metal Infrastructure Provider builds on `elemental-operator` and Cluster API. Operators interact with two CRD groups, both documented here: -πŸ“‹ **Planned** +- `infrastructure.cluster.x-k8s.io/v1beta1` β€” the provider's own resources. +- `elemental.cattle.io/v1beta1` β€” owned by `elemental-operator`; the provider only reads and annotates them, but operators must author the registration/seed objects and inspect inventory objects on every workflow. -API documentation will be available when development begins. +## Custom Resources -## Planned Resources +### Provider resources (`infrastructure.cluster.x-k8s.io/v1beta1`) -- `MetalCluster`: Represents a Kubernetes cluster infrastructure on bare metal -- `MetalMachine`: Represents a bare metal server -- `MetalMachineTemplate`: Template for configuring bare metal servers +| Resource | Description | Documentation | +|---|---|---| +| `BaremetalCluster` | Cluster-level infrastructure: control-plane VIP, endpoint, network type. | [BaremetalCluster](./baremetalcluster.mdx) | +| `BaremetalMachine` | Single infra machine bound to one `MachineInventory`. | [BaremetalMachine](./baremetalmachine.mdx) | +| `BaremetalMachineTemplate` | Template that binds a pool to a `KubeadmControlPlane` or `MachineDeployment`. | [BaremetalMachineTemplate](./baremetalmachinetemplate.mdx) | +| `MachineInventoryPool` | Allowed set of `MachineInventory` names for a cluster. | [MachineInventoryPool](./machineinventorypool.mdx) | + +### Elemental resources (`elemental.cattle.io/v1beta1`) + +| Resource | Description | Documentation | +|---|---|---| +| `MachineRegistration` | Registration URL + first-install cloud-config. Authored once per cluster. | [MachineRegistration](./machineregistration.mdx) | +| `SeedImage` | Triggers ISO build with the registration baked in. | [SeedImage](./seedimage.mdx) | +| `MachineInventory` | Long-lived host identity; consumed by `MachineInventoryPool`. The provider also reads `spec.observedNetwork` (Alauda-fork-only) and writes the `baremetal.alauda.io/*` annotations. | [MachineInventory](./machineinventory.mdx) | + +## Upstream Cluster API resources + +The provider also composes the following upstream resources without changing their schemas. Refer to the [Cluster API reference](../../cluster_api/) for full schemas. + +| Resource | API group | Purpose | +|---|---|---| +| `Cluster` | `cluster.x-k8s.io/v1beta1` | Top-level CAPI resource; carries the bare-metal-specific `cpaas.io/*` annotations. | +| `KubeadmControlPlane` | `controlplane.cluster.x-k8s.io/v1beta1` | Owns control-plane `Machine` lifecycle and kubeadm config. | +| `MachineDeployment` | `cluster.x-k8s.io/v1beta1` | Owns worker `Machine` lifecycle. | +| `KubeadmConfigTemplate` | `bootstrap.cluster.x-k8s.io/v1beta1` | Worker cloud-init `user-data` template. | diff --git a/docs/en/apis/providers/bare-metal/machineinventory.mdx b/docs/en/apis/providers/bare-metal/machineinventory.mdx new file mode 100644 index 00000000..bdae309b --- /dev/null +++ b/docs/en/apis/providers/bare-metal/machineinventory.mdx @@ -0,0 +1,17 @@ +--- +weight: 70 +--- + +# MachineInventory [elemental.cattle.io/v1beta1] + +`MachineInventory` is the long-lived identity object for a physical host. It is created by `elemental-register` after a host boots the seed ISO, and it survives reprovisioning β€” `BaremetalMachine` only borrows it for the lifetime of one Kubernetes node. + +The bare-metal provider reads two parts of this resource: + +- `metadata.name` β€” must appear in a `MachineInventoryPool.spec.inventoryNames` entry for the host to be allocatable. +- `spec.observedNetwork` β€” an Alauda-fork-only field captured during registration; the provider replays it as cloud-init `network-config` v2 on every reprovision, so the host comes back with the same interface/bond/VLAN/IP layout as when it was first inventoried. + +The provider also writes `metadata.annotations` (`baremetal.alauda.io/owner-*`, `baremetal.alauda.io/plan.type`) and updates `status.plan` / `status.conditions` during clean/reprovision. + +{/* cspell:disable-next-line */} + diff --git a/docs/en/apis/providers/bare-metal/machineinventorypool.mdx b/docs/en/apis/providers/bare-metal/machineinventorypool.mdx new file mode 100644 index 00000000..879fe625 --- /dev/null +++ b/docs/en/apis/providers/bare-metal/machineinventorypool.mdx @@ -0,0 +1,8 @@ +--- +weight: 40 +--- + +# MachineInventoryPool [infrastructure.cluster.x-k8s.io/v1beta1] + +{/* cspell:disable-next-line */} + diff --git a/docs/en/apis/providers/bare-metal/machineregistration.mdx b/docs/en/apis/providers/bare-metal/machineregistration.mdx new file mode 100644 index 00000000..1b6fbfb1 --- /dev/null +++ b/docs/en/apis/providers/bare-metal/machineregistration.mdx @@ -0,0 +1,10 @@ +--- +weight: 50 +--- + +# MachineRegistration [elemental.cattle.io/v1beta1] + +`MachineRegistration` is owned by `elemental-operator` and is created once per workload cluster. The bare-metal provider does not mutate it, but operators must author it because its `config` block is baked into the registration ISO and replayed by `elemental-register` on first boot β€” it carries the registration URL, MachineInventoryLabels, and the cloud-init that turns a freshly imaged host into a usable `MachineInventory`. + +{/* cspell:disable-next-line */} + diff --git a/docs/en/apis/providers/bare-metal/seedimage.mdx b/docs/en/apis/providers/bare-metal/seedimage.mdx new file mode 100644 index 00000000..87000be6 --- /dev/null +++ b/docs/en/apis/providers/bare-metal/seedimage.mdx @@ -0,0 +1,10 @@ +--- +weight: 60 +--- + +# SeedImage [elemental.cattle.io/v1beta1] + +`SeedImage` is owned by `elemental-operator`. Creating one references a `MachineRegistration` and triggers the operator to build an ISO with the registration URL embedded; operators download that ISO and boot bare-metal hosts from it. The resulting `MachineInventory` objects are what `MachineInventoryPool` allocates to `BaremetalMachine`. + +{/* cspell:disable-next-line */} + diff --git a/docs/en/create-cluster/bare-metal.mdx b/docs/en/create-cluster/bare-metal.mdx index f1d44eda..9963e0a6 100644 --- a/docs/en/create-cluster/bare-metal.mdx +++ b/docs/en/create-cluster/bare-metal.mdx @@ -1,22 +1,654 @@ --- weight: 40 +title: Creating Clusters on Bare Metal +author: dev@alauda.io +category: howto +queries: + - create baremetal cluster + - deploy kubernetes on bare metal + - baremetal cluster creation guide + - elemental seedimage iso install + - machineinventorypool cluster --- # Creating Clusters on Bare Metal -This document provides instructions for creating Kubernetes clusters on bare-metal servers. +This document explains how to create Kubernetes clusters on physical servers using the bare-metal provider. The workflow is YAML-only β€” there is no Fleet Essentials UI for bare-metal clusters at this time. -## Status +## Prerequisites -πŸ“‹ **Planned** +Before creating clusters, ensure all of the following prerequisites are met. -This provider is in the planning phase. Documentation will be available when development begins. +### 1. Required Plugin Installation -## Planned Features +Install the following plugins on the `global` cluster: -- BMC/IPMI integration for remote management -- In-place upgrade support -- Hardware inventory management -- Network configuration for bare-metal environments +- **Alauda Container Platform Kubeadm Provider** +- **Alauda Container Platform Bare Metal Infrastructure Provider** (umbrella chart that installs both the bare-metal manager and `elemental-operator`) -For more information, see [Bare Metal Provider Overview](../overview/providers/bare-metal.mdx). +See the [Installation Guide](../install/bare-metal.mdx) for details. + +### 2. Image Catalog Confirmed + +The bare-metal provider chart ships an `elemental-image-catalog` ConfigMap that maps `Machine.spec.version` to the `elemental upgrade` image used to (re)provision a node. You do **not** need to create this ConfigMap separately β€” confirm that the target Kubernetes version is present: + +```bash +kubectl -n cpaas-system get configmap elemental-image-catalog -o yaml +``` + +Every value used as `Machine.spec.version` (for both the control plane and worker `MachineDeployment` resources) must appear as a key in this ConfigMap, with the leading `v` preserved. The provider resolves the image at reprovision time by substituting the platform registry address for the registry portion of the entry. If a target version is missing, no `reprovision` plan is written and `BaremetalMachine` ends up in `Failed / Reason=ImageCatalogMiss` until the entry is added. + +### 3. Network Connectivity + +- Every physical host must be able to reach `global.platformUrl` (`elemental-system-agent` registration, plan secret polling). +- Every physical host must be able to pull from the platform registry (`global.registry.address`) for both `elemental install` (during the first boot) and `elemental upgrade` (during every reprovision). +- The control-plane VIP must live in the same Layer-2 broadcast domain as the control-plane node IPs. The `vrid` chosen for the VIP must be unique within that broadcast domain. + +### 4. TPM Decision + +Production hosts should keep `MachineRegistration.spec.config.elemental.registration.emulate-tpm: false` (or remove the field). For PoC and virtual-machine smoke tests, set `emulate-tpm: true` and `emulated-tpm-seed: -1` so registration works without a real TPM. + +### 5. Public Registry Credential (Only When Installing Platform Components Later) + +`public-registry-credential` is **not** required to create a bare-metal cluster. It only becomes necessary when later platform components on the new workload cluster need to pull from a credentialed public registry. If your test scope ends at cluster + node Ready, you can ignore this prerequisite. + +--- + +## Cluster Creation Workflow + +When using YAML, the workflow proceeds through five steps. Every step must be applied in the `cpaas-system` namespace. + +```text +1. MachineRegistration + SeedImage ─► Build the bootable ISO +2. Boot the ISO on each host ─► elemental install + register MachineInventory +3. MachineInventoryPool (per role) ─► Pre-declare the allowed inventories +4. BaremetalCluster + BaremetalMachineTemplate + KubeadmControlPlane + Cluster +5. KubeadmConfigTemplate + worker BaremetalMachineTemplate + MachineDeployment +``` + +:::warning +**Important Namespace Requirement** + +All bare-metal resources must be applied in the `cpaas-system` namespace. The provider and `elemental-operator` only reconcile objects in that namespace. +::: + +:::warning +**Workload Cluster Naming** + +The workload `cluster-name` **must not** be `global`. That name is reserved for the `global` cluster, and reusing it causes the workload cluster's resources to collide with `global` cluster resources in `cpaas-system`. As a convention, keep the CAPI `Cluster` and `BaremetalCluster` named exactly ``, and prefix dependent resources (`KubeadmControlPlane`, `KubeadmConfigTemplate`, `MachineDeployment`, machine templates, pools, registrations) with `-`. +::: + +### Resolving Placeholder Values \{#resolving-placeholders} + +The example manifests below use `` syntax for environment-specific values: + +| Placeholder | Source of truth | How to retrieve | +|---|---|---| +| `` | `elemental-image-catalog` ConfigMap key (with leading `v`, for example `v1.33.7-2`). | `kubectl -n cpaas-system get cm elemental-image-catalog -o yaml` | +| `` | Same ConfigMap entry, with `-iso` appended to the repository. Tag/digest matches the catalog value. | See the [SeedImage step](#step-1-machineregistration-seedimage). | +| `` | Platform registry (same value as `global.registry.address` on the install chart). | `kubectl get cluster global -n cpaas-system -o jsonpath='{.metadata.annotations.cpaas\.io/registry-address}'` (when one exists). | +| `` / `` / `` | Operator-supplied. The VIP must be free in the control-plane Layer-2 domain; `` must be unique in that domain. | n/a | +| `` / `` | Component versions baked into the bare-metal base image for ``. | See [OS Support Matrix](../overview/os-support-matrix.mdx). | +| `` | Operator-supplied. Written into `/etc/resolv.conf` by both `MachineRegistration.config.cloud-config` and `SeedImage.cloud-config`. | n/a | +| `` | Operator-supplied OpenSSH public key β€” required for any interactive debugging on the node. | n/a | +| `` | Allocated by `elemental-operator` after the host boots the ISO and registers. Cannot be predicted before registration. | `kubectl -n cpaas-system get machineinventories.elemental.cattle.io` | +| `` / `` / `` | Operator-supplied. Must not overlap with the host network, the `global` cluster's CIDRs, or any other CAPI cluster on the same `global`. | n/a | + +### Step 1: Build the SeedImage and Register Hosts \{#step-1-machineregistration-seedimage} + +Create a `MachineRegistration` that describes the registration URL and first-install cloud-config, and a `SeedImage` that points `elemental-operator` at the matching ISO base image. + +`SeedImage.spec.baseImage` is derived from the image catalog entry for the target Kubernetes version: take the catalog repository, append `-iso`, and keep the same tag or digest. For example, if `elemental-image-catalog` resolves `v1.33.7-2` to `/tkestack/baremetal-base-image:v0.0.0-beta-1.33.7-2`, then `SeedImage.spec.baseImage` is `/tkestack/baremetal-base-image-iso:v0.0.0-beta-1.33.7-2`. + +```yaml title="01-machineregistration-seedimage.yaml" +apiVersion: elemental.cattle.io/v1beta1 +kind: MachineRegistration +metadata: + name: -registration + namespace: cpaas-system + labels: + app.kubernetes.io/part-of: cluster-api-provider-baremetal +spec: + # ${...} placeholders are expanded by elemental-register from SMBIOS data. + # System UUID is preferred over Serial Number because it is more reliable + # across virtualised test environments. + machineName: "-${System Information/UUID}" + machineInventoryLabels: + app.kubernetes.io/part-of: cluster-api-provider-baremetal + pool.baremetal.alauda.io/eligible: "true" + elemental.cattle.io/serial-number: "${System Information/Serial Number}" + machineInventoryAnnotations: + elemental.cattle.io/machine-uuid: "${System Information/UUID}" + config: + elemental: + install: + reboot: false + snapshotter: + type: btrfs + maxSnaps: 4 + registration: + # Remove emulate-tpm for production hardware with a real TPM. + emulate-tpm: true + emulated-tpm-seed: -1 +--- +apiVersion: elemental.cattle.io/v1beta1 +kind: SeedImage +metadata: + name: -registration-iso + namespace: cpaas-system + labels: + app.kubernetes.io/part-of: cluster-api-provider-baremetal +spec: + type: iso + baseImage: # See the table above for derivation. + registrationRef: + apiVersion: elemental.cattle.io/v1beta1 + kind: MachineRegistration + name: -registration + namespace: cpaas-system + targetPlatform: linux/amd64 + size: 20Gi + cleanupAfterMinutes: 60 + cloud-config: + write_files: + - path: /etc/resolv.conf + permissions: "0644" + content: | + nameserver + - path: /etc/elemental/config.d/partitions.yaml + permissions: "0644" + content: | + install: + partitions: + state: + size: 20480 +``` + +Apply the manifest and wait for the SeedImage build to finish: + +```bash +kubectl apply -f 01-machineregistration-seedimage.yaml +kubectl -n cpaas-system get seedimage -registration-iso -w +``` + +When `status.state` reaches `Completed`, fetch the download URL and ISO checksum: + +```bash +kubectl -n cpaas-system get seedimage -registration-iso \ + -o jsonpath='{.status.downloadURL}{"\n"}' +kubectl -n cpaas-system get seedimage -registration-iso \ + -o jsonpath='{.status.checksumURL}{"\n"}' +``` + +Boot every target host from this ISO. `elemental-register` runs first (creates the `MachineInventory` and uploads `observedNetwork`), then `elemental install` writes the on-disk OS. After install completes, the host stays available for plan execution. + +Confirm registration: + +```bash +kubectl -n cpaas-system get machineinventories.elemental.cattle.io +``` + +Every inventory you intend to use must: + +- Show `Ready=True`. +- Have a non-empty `status.plan.secretRef.name`. +- Have a `spec.observedNetwork` that matches the host's expected NIC (only required when you want the install-time IP to survive across reprovisions). + +Record the exact `MachineInventory` names β€” they are referenced by name in the next step. + +### Step 2: Create `MachineInventoryPool` Resources \{#step-2-pools} + +Create one pool per role. The pool reconciler validates that every member exists, computes capacity counters, and writes the `baremetal.alauda.io/pool=` annotation onto the inventory. + +```yaml title="02-machineinventorypool.yaml" +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: MachineInventoryPool +metadata: + name: -control-plane-pool + namespace: cpaas-system +spec: + clusterName: + machineInventories: + - name: + hostname: # Optional; falls back to the inventory name. + - name: + hostname: + - name: + hostname: +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: MachineInventoryPool +metadata: + name: -worker-pool + namespace: cpaas-system +spec: + clusterName: + machineInventories: + - name: + hostname: + - name: + - name: +``` + +**Key parameters:** + +| Parameter | Type | Description | Required | +|---|---|---|---| +| `.spec.clusterName` | string | The workload cluster name this pool serves. A `MachineInventory` must not appear in two active pools. | Yes | +| `.spec.machineInventories[].name` | string | Exact name of a registered `MachineInventory`. | Yes | +| `.spec.machineInventories[].hostname` | string | Hostname applied during reprovision. Defaults to the inventory name when omitted. | No | + +Apply and verify: + +```bash +kubectl apply -f 02-machineinventorypool.yaml +kubectl -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io +``` + +A healthy pool reports `Ready=True`, `total = len(spec.machineInventories)`, and `available = total - allocated - preparing - reprovisioning - unavailable`. Inventories listed in `spec.machineInventories` that fail validation (missing, plan secret missing, Ready=False) raise the pool's `unavailable` counter and surface in the `MembersValid` condition. + +Size the control-plane pool to at least `KubeadmControlPlane.spec.replicas`. Size the worker pool to at least `MachineDeployment.spec.replicas`. For rolling upgrades the pool must hold the entire replica count β€” the provider uses delete-then-add semantics from the same pool, never both at once. + +### Step 3: Create the Control-Plane Cluster Resources \{#step-3-control-plane} + +Create the `BaremetalCluster` (declares the control-plane VIP), the control-plane `BaremetalMachineTemplate` (points at the control-plane pool), the `KubeadmControlPlane` (replicas + kubeadm config), and the CAPI `Cluster`. + +:::tip +**Full Configuration Reference** + +The example below uses a minimal `KubeadmControlPlane`. For the full hardening profile recommended in production β€” admission, audit, kubelet patches, encryption provider β€” see [Complete KubeadmControlPlane Configuration](#complete-kubeadmcontrolplane-configuration) in the Appendix. +::: + +```yaml title="03-cluster.yaml" +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: BaremetalCluster +metadata: + name: + namespace: cpaas-system +spec: + controlPlaneLoadBalancer: + type: Internal # External: skip alive deployment, supply your own LB. + host: + port: + vrid: + networkType: kube-ovn # Phase-1 metadata only. +--- +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: BaremetalMachineTemplate +metadata: + name: -control-plane-template + namespace: cpaas-system +spec: + template: + spec: + machineInventoryPoolRef: + name: -control-plane-pool +--- +apiVersion: controlplane.cluster.x-k8s.io/v1beta1 +kind: KubeadmControlPlane +metadata: + name: -control-plane + namespace: cpaas-system +spec: + replicas: 3 + version: + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 # Bare-metal does not over-provision; replace one at a time. + machineTemplate: + nodeDrainTimeout: 5m + nodeVolumeDetachTimeout: 5m + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: BaremetalMachineTemplate + name: -control-plane-template + kubeadmConfigSpec: + users: + - name: boot + sudo: ALL=(ALL) NOPASSWD:ALL + shell: /bin/bash + sshAuthorizedKeys: + - "" + clusterConfiguration: + imageRepository: /tkestack + dns: + imageTag: + etcd: + local: + imageTag: + apiServer: + extraArgs: + profiling: "false" + tls-min-version: VersionTLS12 + controllerManager: + extraArgs: + profiling: "false" + tls-min-version: VersionTLS12 + scheduler: + extraArgs: + profiling: "false" + tls-min-version: VersionTLS12 + initConfiguration: + nodeRegistration: + kubeletExtraArgs: + node-labels: "kube-ovn/role=master" + joinConfiguration: + nodeRegistration: + kubeletExtraArgs: + node-labels: "kube-ovn/role=master" +--- +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: + namespace: cpaas-system + annotations: + capi.cpaas.io/resource-group-version: infrastructure.cluster.x-k8s.io/v1beta1 + capi.cpaas.io/resource-kind: BaremetalCluster + cpaas.io/kube-ovn-join-cidr: + cpaas.io/sentry-deploy-type: Baremetal + cpaas.io/alb-address-type: ClusterAddress + labels: + cluster-type: ProviderBaremetal +spec: + clusterNetwork: + pods: + cidrBlocks: + - + services: + cidrBlocks: + - + controlPlaneRef: + apiVersion: controlplane.cluster.x-k8s.io/v1beta1 + kind: KubeadmControlPlane + name: -control-plane + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: BaremetalCluster + name: +``` + +**Cluster annotations.** The bare-metal provider relies on a small set of `Cluster` annotations during reconcile. Authoritative ones the operator must set: + +| Annotation | Required | Value source | Purpose | +|---|---|---|---| +| `capi.cpaas.io/resource-group-version` | Yes | Literal `infrastructure.cluster.x-k8s.io/v1beta1` | CAPI infrastructure binding. | +| `capi.cpaas.io/resource-kind` | Yes | Literal `BaremetalCluster` | CAPI infrastructure binding. | +| `cpaas.io/kube-ovn-join-cidr` | Yes | Operator-chosen `/16` CIDR that does not overlap with pods/services or any other cluster. | Kube-OVN inter-node tunnel. | +| `cpaas.io/sentry-deploy-type` | Yes | Literal `Baremetal`. | Marks the cluster for the bare-metal deploy profile. | +| `cpaas.io/alb-address-type` | Yes | Literal `ClusterAddress`. | ALB address mode used on bare-metal clusters. | + +**`BaremetalCluster` parameters:** + +| Parameter | Type | Description | Required | +|---|---|---|---| +| `.spec.controlPlaneLoadBalancer.type` | string (`Internal` / `External`) | `Internal` deploys `alive` static pods on the control-plane nodes and manages the VIP. `External` skips `alive` and assumes an external LB already terminates `:`. | No (defaults to `Internal`) | +| `.spec.controlPlaneLoadBalancer.host` | string | Control-plane VIP. Backfilled into `.spec.controlPlaneEndpoint.host` when the endpoint is left empty. | Yes | +| `.spec.controlPlaneLoadBalancer.port` | int (1–65535) | Control-plane port. Typically `6443`. | Yes | +| `.spec.controlPlaneLoadBalancer.vrid` | int (0–255) | `keepalived` `virtual_router_id`. Must be unique in the control-plane Layer-2 domain. Only consumed when `type=Internal`. | When `type=Internal` | +| `.spec.controlPlaneEndpoint` | object | API server endpoint exposed to CAPI. Once set, must not change. Leave empty to let the reconciler backfill it from `controlPlaneLoadBalancer`. | No | +| `.spec.networkType` | string | CNI hint. `kube-ovn` today (Phase-1 metadata only). | No | + +Apply and watch: + +```bash +kubectl apply -f 03-cluster.yaml +kubectl -n cpaas-system get clusters.cluster.x-k8s.io +kubectl -n cpaas-system get kubeadmcontrolplanes.controlplane.cluster.x-k8s.io +kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io -w +``` + +Each new control-plane `BaremetalMachine` advances `Pending β†’ Allocated β†’ Reprovisioning β†’ Running`. Watch: + +- `BaremetalMachine.status.machineInventoryRef.name` β€” which inventory was picked. +- `BaremetalMachine.status.planSecretRef.name` β€” plan secret being driven. The secret carries `baremetal.alauda.io/plan.type=reprovision`. +- `MachineInventory.status.plan.state` β€” `Applied` once the host completes `cloud-init clean`, `elemental upgrade`, reboot, and `kubeadm init`/`join`. +- `BaremetalCluster.status.conditions[EndpointReady]` β€” true once the VIP is reachable. + +:::warning +The bare-metal provider does not support single-node control planes. Provision at least three control-plane replicas (`KubeadmControlPlane.spec.replicas: 3`) so that `alive` can arbitrate the VIP and etcd retains quorum. +::: + +### Step 4: Deploy Worker Nodes + +After the control plane is Ready, create the worker `BaremetalMachineTemplate`, the worker `KubeadmConfigTemplate`, and the `MachineDeployment`. The full worker YAML and parameter table are in [Managing Nodes on Bare Metal β†’ Worker Node Deployment](../manage-nodes/bare-metal.mdx#worker-node-deployment). + +--- + +## Cluster Verification + +### Using kubectl + +```bash +# Cluster status +kubectl -n cpaas-system get cluster + +# Control plane progress +kubectl -n cpaas-system get kubeadmcontrolplane -control-plane + +# Per-machine state +kubectl -n cpaas-system get machines.cluster.x-k8s.io +kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io + +# Pool counters +kubectl -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io + +# Workload cluster Nodes (use the workload kubeconfig) +kubectl get nodes -o wide +``` + +### Expected Results + +A successfully created cluster shows: + +- `Cluster.status.conditions[Ready]=True`. +- `KubeadmControlPlane` replicas all `Ready`. +- Every `BaremetalMachine.status.phase=Running` and `status.ready=true`. +- Every used `MachineInventory.status.plan.state=Applied` with `baremetal.alauda.io/plan.type=reprovision` on its plan secret. +- `MachineInventoryPool.status` satisfies `available + allocated + preparing + reprovisioning + unavailable = total`. +- Kubernetes Nodes Ready. + +--- + +## Common Failure Modes + +| Symptom | Likely cause | Where to look | +|---|---|---| +| `BaremetalMachine` stuck in `Pending`, `InventoryAllocated=False / Reason=PoolMissing` | `BaremetalMachineTemplate.spec.template.spec.machineInventoryPoolRef.name` points at a non-existent pool. | Pool name + namespace. | +| `BaremetalMachine` stuck in `Pending`, `InventoryAllocated=False / Reason=PoolExhausted` | Pool has no `Available` inventory left. | `MachineInventoryPool.status.available`; allocation annotations on each member. | +| `BaremetalMachine` stuck after allocation, `BootstrapReady=False / Reason=BootstrapWaiting` | KubeadmConfig has not produced a bootstrap data secret yet. | `Machine.spec.bootstrap.dataSecretName`. | +| `BaremetalMachine` in `Failed`, `ImageResolved=False / Reason=ImageCatalogMiss` | The target `Machine.spec.version` is not a key in `elemental-image-catalog`. | `kubectl -n cpaas-system get cm elemental-image-catalog -o yaml`. | +| `ImageResolved=False / Reason=ImageRegistryMissing` | Neither the optional `cpaas.io/registry-address` annotation nor the platform registry-credential Secret resolves a registry address. | Cluster annotations; platform registry-credential Secret. | +| Reprovision never completes; `MachineInventory.status.plan.state=Failed` | `elemental upgrade` failed (registry unreachable, TLS, disk full). | Plan secret `failed-output` field; host serial console. | +| `BaremetalCluster` Ready, but the VIP is not reachable | `vrid` collides with another cluster; control-plane VIP not in the L2 domain; firewall blocking VRRP. | `kubectl -n kube-system get pod -l app=alive`, `ip addr show`, `ipvsadm -Ln` on a control-plane host. | + +For the full operator-side state machine reference (every condition reason and recovery action), see [Provider Overview β†’ clean / reprovision plans](../overview/providers/bare-metal.mdx#clean-reprovision). + +--- + +## Next Steps + +After creating a cluster: + +- [Manage Nodes](../manage-nodes/bare-metal.mdx) +- [Upgrade Cluster](../upgrade-cluster/bare-metal.mdx) + +--- + +## Appendix + +### Complete `KubeadmControlPlane` Configuration \{#complete-kubeadmcontrolplane-configuration} + +The hardened configuration recommended for production bare-metal clusters β€” admission control, audit policy, kubelet patches, encryption provider, and IPv6 bind addresses. Substitute the placeholders from the table in [Resolving Placeholder Values](#resolving-placeholders). + +```yaml +apiVersion: controlplane.cluster.x-k8s.io/v1beta1 +kind: KubeadmControlPlane +metadata: + name: -control-plane + namespace: cpaas-system +spec: + replicas: 3 + version: + rolloutStrategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + machineTemplate: + nodeDrainTimeout: 5m + nodeVolumeDetachTimeout: 5m + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: BaremetalMachineTemplate + name: -control-plane-template + kubeadmConfigSpec: + users: + - name: boot + sudo: ALL=(ALL) NOPASSWD:ALL + shell: /bin/bash + sshAuthorizedKeys: + - "" + files: + - path: /etc/kubernetes/admission/psa-config.yaml + owner: "root:root" + permissions: "0644" + content: | + apiVersion: apiserver.config.k8s.io/v1 + kind: AdmissionConfiguration + plugins: + - name: PodSecurity + configuration: + apiVersion: pod-security.admission.config.k8s.io/v1 + kind: PodSecurityConfiguration + defaults: + enforce: "privileged" + enforce-version: "latest" + audit: "baseline" + audit-version: "latest" + warn: "baseline" + warn-version: "latest" + exemptions: + usernames: [] + runtimeClasses: [] + namespaces: + - kube-system + - cpaas-system + - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json + owner: "root:root" + permissions: "0644" + content: | + { + "apiVersion": "kubelet.config.k8s.io/v1beta1", + "kind": "KubeletConfiguration", + "protectKernelDefaults": true, + "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt", + "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key", + "streamingConnectionIdleTimeout": "5m", + "clientCAFile": "/etc/kubernetes/pki/ca.crt" + } + - path: /etc/kubernetes/audit/policy.yaml + owner: "root:root" + permissions: "0644" + content: | + apiVersion: audit.k8s.io/v1 + kind: Policy + omitStages: + - "RequestReceived" + rules: + - level: None + users: + - system:kube-controller-manager + - system:kube-scheduler + - system:serviceaccount:kube-system:endpoint-controller + verbs: ["get", "update"] + namespaces: ["kube-system"] + resources: + - group: "" + resources: ["endpoints"] + - level: None + nonResourceURLs: + - /healthz* + - /version + - /swagger* + - level: None + resources: + - group: "" + resources: ["events"] + - level: None + verbs: ["get", "list", "watch"] + - level: None + resources: + - group: "coordination.k8s.io" + resources: ["leases"] + - level: None + resources: + - group: "authorization.k8s.io" + resources: ["subjectaccessreviews", "selfsubjectaccessreviews"] + - group: "authentication.k8s.io" + resources: ["tokenreviews"] + - level: Metadata + resources: + - group: "" + resources: ["secrets", "configmaps"] + - level: RequestResponse + resources: + - group: "" + - group: "apps" + - group: "rbac.authorization.k8s.io" + - group: "storage.k8s.io" + - group: "networking.k8s.io" + - level: Metadata + postKubeadmCommands: + - chmod 600 /var/lib/kubelet/config.yaml + clusterConfiguration: + imageRepository: /tkestack + dns: + imageTag: + etcd: + local: + imageTag: + apiServer: + extraArgs: + audit-log-format: json + audit-log-maxage: "30" + audit-log-maxbackup: "10" + audit-log-maxsize: "200" + profiling: "false" + audit-log-mode: batch + audit-log-path: /etc/kubernetes/audit/audit.log + audit-policy-file: /etc/kubernetes/audit/policy.yaml + tls-cipher-suites: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384" + admission-control-config-file: /etc/kubernetes/admission/psa-config.yaml + tls-min-version: VersionTLS12 + kubelet-certificate-authority: /etc/kubernetes/pki/ca.crt + extraVolumes: + - name: vol-dir-0 + hostPath: /etc/kubernetes + mountPath: /etc/kubernetes + pathType: Directory + controllerManager: + extraArgs: + bind-address: "::" + profiling: "false" + tls-min-version: VersionTLS12 + scheduler: + extraArgs: + bind-address: "::" + tls-min-version: VersionTLS12 + profiling: "false" + initConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + node-labels: "kube-ovn/role=master" + joinConfiguration: + patches: + directory: /etc/kubernetes/patches + nodeRegistration: + kubeletExtraArgs: + node-labels: "kube-ovn/role=master" +``` + +Worker bootstrap is symmetric β€” see [Managing Nodes on Bare Metal β†’ Bootstrap Template](../manage-nodes/bare-metal.mdx#step-3-configure-bootstrap-template) for the worker `KubeadmConfigTemplate`. diff --git a/docs/en/create-cluster/index.mdx b/docs/en/create-cluster/index.mdx index 0295b539..76c9afdf 100644 --- a/docs/en/create-cluster/index.mdx +++ b/docs/en/create-cluster/index.mdx @@ -31,7 +31,7 @@ Select your platform for detailed cluster creation instructions: | [Huawei DCS](./huawei-dcs.mdx) | Huawei Datacenter Virtualization Solution | βœ… Available | | [Huawei Cloud Stack](./huawei-cloud-stack.mdx) | Huawei Cloud Stack | βœ… Available | | [VMware vSphere](./vmware-vsphere/) | VMware vSphere virtualization platform | βœ… Available | -| [Bare Metal](./bare-metal.mdx) | Bare-metal servers without virtualization | πŸ“‹ Planned | +| [Bare Metal](./bare-metal.mdx) | Bare-metal servers without virtualization | βœ… Available (YAML only) | ## Next Steps diff --git a/docs/en/install/bare-metal.mdx b/docs/en/install/bare-metal.mdx index 2961e12d..c9cf3303 100644 --- a/docs/en/install/bare-metal.mdx +++ b/docs/en/install/bare-metal.mdx @@ -6,10 +6,72 @@ weight: 40 This document describes how to install the Bare Metal Infrastructure Provider for Immutable Infrastructure. -## Status +## Prerequisites -πŸ“‹ **Planned** +Before installing the provider, ensure you have: -This provider is in the planning phase. Documentation will be available when development begins. +- Access to the `global` cluster. +- Access to Customer Portal for downloading plugins. -For more information about the Bare Metal platform, see [Provider Overview](../overview/providers/bare-metal.mdx). +## Downloading + +:::info +The bare-metal workflow requires two plugins from Customer Portal: +1. **Alauda Container Platform Kubeadm Provider** +2. **Alauda Container Platform Bare Metal Infrastructure Provider** (umbrella chart that ships the `cluster-api-provider-baremetal-manager` and `elemental-operator` subcharts together) +::: + +Download both plugins from the Customer Portal. + +## Uploading + +For detailed instructions on uploading packages, refer to . + +## Installing + +For detailed instructions on installing cluster plugins, refer to . + +The umbrella chart installs both the bare-metal provider manager and `elemental-operator` in one step. The image catalog ConfigMap is created by the chart β€” you do not need to apply it separately when creating a workload cluster. + +## Verifying Installation + +After installing the plugins, verify that the controllers and CRDs are present: + +```bash +# Provider manager + elemental-operator should both be Running +kubectl -n cpaas-system get pods | grep -E 'baremetal|elemental' + +# Bare-metal CRDs +kubectl get crd | grep -E '^(baremetalclusters|baremetalmachines|baremetalmachinetemplates|machineinventorypools)\.infrastructure\.cluster\.x-k8s\.io$' + +# Elemental CRDs (shipped by the elemental subchart) +kubectl get crd | grep -E '^(machineinventories|machineregistrations|seedimages)\.elemental\.cattle\.io$' + +# Image catalog ConfigMap +kubectl -n cpaas-system get configmap elemental-image-catalog +``` + +Expected output includes: + +- `baremetalclusters.infrastructure.cluster.x-k8s.io` +- `baremetalmachines.infrastructure.cluster.x-k8s.io` +- `baremetalmachinetemplates.infrastructure.cluster.x-k8s.io` +- `machineinventorypools.infrastructure.cluster.x-k8s.io` +- `machineinventories.elemental.cattle.io` +- `machineregistrations.elemental.cattle.io` +- `seedimages.elemental.cattle.io` + +Confirm that `elemental-image-catalog` contains the Kubernetes versions you intend to deploy: + +```bash +kubectl -n cpaas-system get configmap elemental-image-catalog -o yaml +``` + +Every `Machine.spec.version` used by a cluster β€” both at creation time and at upgrade time β€” must appear as a key in this ConfigMap. + +## Next Steps + +After installing the provider, you can proceed to: + +- [Create a Cluster](../create-cluster/bare-metal.mdx) +- Review [Provider Overview](../overview/providers/bare-metal.mdx) for the pool model and plan lifecycle. diff --git a/docs/en/install/index.mdx b/docs/en/install/index.mdx index db49c6e5..3749c933 100644 --- a/docs/en/install/index.mdx +++ b/docs/en/install/index.mdx @@ -36,4 +36,4 @@ Select your platform for detailed installation instructions: - [Huawei DCS](./huawei-dcs.mdx) - [Huawei Cloud Stack](./huawei-cloud-stack.mdx) - [VMware vSphere](./vmware-vsphere.mdx) -- Bare Metal (Planned) +- [Bare Metal](./bare-metal.mdx) diff --git a/docs/en/manage-nodes/bare-metal.mdx b/docs/en/manage-nodes/bare-metal.mdx index bb252063..de40603a 100644 --- a/docs/en/manage-nodes/bare-metal.mdx +++ b/docs/en/manage-nodes/bare-metal.mdx @@ -4,10 +4,392 @@ weight: 40 # Managing Nodes on Bare Metal -This document provides instructions for managing worker nodes on bare metal servers. +This document explains how to deploy worker nodes, scale them up and down, replace specific machines, and recover failed nodes on the bare-metal provider. Node management uses Cluster API `Machine` resources orchestrated through `MachineDeployment`; the provider binds each `Machine` to one `MachineInventory` from a pool and drives the host through `clean` / `reprovision` plans. -## Status +## Prerequisites -πŸ“‹ **Planned** +:::warning +**Important Prerequisites** -Documentation will be available when development begins. +- The control plane must already be running. See [Create Cluster](../create-cluster/bare-metal.mdx). +- The worker `MachineInventoryPool` must contain at least as many `Available` inventories as the target replica count, plus the headroom required by the rollout strategy. +- The target `Machine.spec.version` must be a key in the `elemental-image-catalog` ConfigMap. +::: + +:::info +**Configuration Guidelines** + +When working with the configurations in this document: + +- Only modify values enclosed in `<>` brackets. +- Replace placeholder values with your environment-specific settings. +- Preserve all other default configurations unless explicitly required. +::: + +## Overview + +The four resources that compose a worker node group: + +1. **`MachineInventoryPool`** (`-worker-pool`) β€” the allowed set of `MachineInventory` names. Already created in [Create Cluster β†’ Step 2](../create-cluster/bare-metal.mdx#step-2-pools). +2. **`BaremetalMachineTemplate`** (`-worker-template`) β€” points at the worker pool. CAPI requires that this template be replaced (new `metadata.name`) every time the underlying pool reference or allocation policy changes. +3. **`KubeadmConfigTemplate`** (`-worker-bootstrap`) β€” cloud-init `user-data` for `kubeadm join`. The bare-metal provider normalizes this user-data at reprovision time (hostname, `provider-id`, `criSocket`); operators should not pre-fill those fields. +4. **`MachineDeployment`** β€” controls replica count, version, rollout strategy. + +The Cluster API contract for templates: the running `Machine` set keeps an in-memory snapshot of the previous template, so editing a template in place does **not** trigger a rollout. Worker upgrades create a new `BaremetalMachineTemplate` and patch `MachineDeployment.spec.template.spec.infrastructureRef.name` to point at it. + +## Worker Node Deployment \{#worker-node-deployment} + +### Step 1: Confirm the Worker Pool Capacity \{#step-1-pool-capacity} + +```bash +kubectl -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io -worker-pool +``` + +`status.available` must be at least the target replica count. If you need additional capacity, register more hosts (boot the `SeedImage` ISO on them β€” see [Create Cluster β†’ Step 1](../create-cluster/bare-metal.mdx#step-1-machineregistration-seedimage)) and add their `MachineInventory` names to `spec.machineInventories`. + +### Step 2: Configure the Worker `BaremetalMachineTemplate` + +```yaml +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: BaremetalMachineTemplate +metadata: + name: -worker-template + namespace: cpaas-system +spec: + template: + spec: + machineInventoryPoolRef: + name: -worker-pool +``` + +**Key parameters:** + +| Parameter | Type | Description | Required | +|---|---|---|---| +| `.spec.template.spec.machineInventoryPoolRef.name` | string | Worker pool name (same namespace). Immutable once the template is created. | Yes | + +`allocationPolicy` is reserved for future expansion. The provider currently treats every pool as `Ordered` β€” it picks the first `Available` inventory in declaration order. + +### Step 3: Configure the Bootstrap Template \{#step-3-configure-bootstrap-template} + +```yaml +apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 +kind: KubeadmConfigTemplate +metadata: + name: -worker-bootstrap + namespace: cpaas-system +spec: + template: + spec: + users: + - name: boot + sudo: ALL=(ALL) NOPASSWD:ALL + shell: /bin/bash + sshAuthorizedKeys: + - "" + files: + - path: /etc/kubernetes/patches/kubeletconfiguration0+strategic.json + owner: "root:root" + permissions: "0644" + content: | + { + "apiVersion": "kubelet.config.k8s.io/v1beta1", + "kind": "KubeletConfiguration", + "protectKernelDefaults": true, + "staticPodPath": null, + "tlsCertFile": "/etc/kubernetes/pki/kubelet.crt", + "tlsPrivateKeyFile": "/etc/kubernetes/pki/kubelet.key", + "streamingConnectionIdleTimeout": "5m", + "clientCAFile": "/etc/kubernetes/pki/ca.crt" + } + joinConfiguration: + patches: + directory: /etc/kubernetes/patches +``` + +The provider applies a minimal normalization to bootstrap user-data before writing it into the `reprovision` plan, so leave the following fields out of the template: + +- **Hostname / FQDN** β€” set automatically from `MachineInventoryPool.spec.machineInventories[].hostname` (or the inventory name when omitted). +- **`kubeletExtraArgs.provider-id`** β€” set automatically to `baremetal:///`. +- **`nodeRegistration.criSocket`** β€” set automatically to `unix:///var/run/containerd/containerd.sock` when unset. + +The normalized user-data is also written back into the bootstrap secret under `data["resolved-value"]` for debugging; the original `data["value"]` is left untouched. + +### Step 4: Configure the `MachineDeployment` + +```yaml +apiVersion: cluster.x-k8s.io/v1beta1 +kind: MachineDeployment +metadata: + name: -workers + namespace: cpaas-system +spec: + clusterName: + replicas: 3 + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 # Bare-metal pools cannot over-provision. + maxUnavailable: 1 + selector: + matchLabels: {} + template: + metadata: + labels: + cluster.x-k8s.io/cluster-name: + pool.name: -workers + spec: + clusterName: + version: + nodeDrainTimeout: 5m + nodeVolumeDetachTimeout: 5m + bootstrap: + configRef: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfigTemplate + name: -worker-bootstrap + namespace: cpaas-system + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: BaremetalMachineTemplate + name: -worker-template + namespace: cpaas-system +``` + +**Key parameters:** + +| Parameter | Type | Description | Required | +|---|---|---|---| +| `.spec.clusterName` | string | Target cluster name. | Yes | +| `.spec.replicas` | int | Number of worker nodes. Must satisfy `replicas ≀ MachineInventoryPool.status.available + status.allocated` (for the worker pool). | Yes | +| `.spec.template.spec.version` | string | Worker Kubernetes version. Must be a key in `elemental-image-catalog`. May differ from the control-plane version while respecting the standard kubelet skew policy. | Yes | +| `.spec.strategy.rollingUpdate.maxSurge` | int | Bare-metal does not over-provision β€” keep `0`. The provider has no spare physical host to bring up an extra node first. | Yes | +| `.spec.strategy.rollingUpdate.maxUnavailable` | int | Must be `> 0` when `maxSurge=0`. Workers are recycled one-by-one within this budget. | Yes | + +Apply and watch: + +```bash +kubectl apply -f workers.yaml +kubectl -n cpaas-system get machinedeployments.cluster.x-k8s.io +kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io -w +kubectl get nodes -o wide # workload cluster +``` + +--- + +## Node Management Operations + +### Scaling Worker Nodes + +#### Adding Worker Nodes + +**Use Case**: Increase cluster capacity. + +**Prerequisites**: + +- The worker pool has enough `Available` inventories. If not, register the additional hosts (boot the SeedImage ISO and confirm the new `MachineInventory` objects) and append them to `MachineInventoryPool.spec.machineInventories[]`. + +**Procedure**: + +1. **Check current state** + + ```bash + kubectl -n cpaas-system get machines.cluster.x-k8s.io \ + -l cluster.x-k8s.io/deployment-name=-workers + kubectl -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io \ + -worker-pool + ``` + + Confirm that `MachineInventoryPool.status.available` is high enough for the planned increment. + +2. **Extend the worker pool when more inventories are needed** + + ```bash + kubectl -n cpaas-system edit machineinventorypool -worker-pool + ``` + + Append the new `MachineInventory` names β€” they must already be registered and Ready. Save and exit; the pool reconciler picks up the change and increases `status.total` and `status.available`. + +3. **Scale up the `MachineDeployment`** + + ```bash + kubectl -n cpaas-system patch machinedeployment -workers \ + --type='json' -p='[{"op":"replace","path":"/spec/replicas","value":}]' + ``` + +4. **Monitor** + + ```bash + kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io -w + kubectl get nodes # workload cluster + ``` + + New `BaremetalMachine` objects advance `Pending β†’ Allocated β†’ Reprovisioning β†’ Running`; the pool's `available` counter decreases as new nodes are bound. + +#### Removing Worker Nodes + +Two strategies are supported, identical in shape to the upstream Cluster API contract: + +| Strategy | When to use | +|---|---| +| **Random removal** | Any node can be removed β€” for example, a temporary capacity reduction. | +| **Targeted removal** | A specific physical host should be released (hardware maintenance, replacement, IP recovery). | + +:::info +**Inventory recycling** + +A `clean` plan stops kubelet, clears CRI workload, and stops containerd. It does **not** wipe Kubernetes persistent directories or reset the OS β€” that work happens later, when the same or a different inventory is picked for a new `BaremetalMachine` and runs the `reprovision` plan. Until then, the inventory returns to `Available` and may be allocated to another cluster. +::: + +##### Random Removal + +```bash +kubectl -n cpaas-system patch machinedeployment -workers \ + --type='json' -p='[{"op":"replace","path":"/spec/replicas","value":}]' +``` + +CAPI selects machines for deletion in its standard order. Each selected `BaremetalMachine` moves to `Preparing`, the provider writes a `clean` plan, and once the plan reports `Applied`, the inventory returns to `Available`. + +##### Targeted Removal + +1. **Identify the machines** + + ```bash + kubectl -n cpaas-system get machines.cluster.x-k8s.io \ + -l cluster.x-k8s.io/deployment-name=-workers + ``` + +2. **Annotate the target machines** + + ```bash + kubectl -n cpaas-system patch machine \ + --type='merge' -p='{"metadata":{"annotations":{"cluster.x-k8s.io/delete-machine":"true"}}}' + ``` + + Repeat for every machine you want to remove. + +3. **Scale down by exactly the number of annotated machines** + + ```bash + kubectl -n cpaas-system patch machinedeployment -workers \ + --type='json' -p='[{"op":"replace","path":"/spec/replicas","value":}]' + ``` + + Reducing by fewer leaves annotated machines in place; reducing by more sends random machines through the `clean` plan as well. + +4. **Verify cleanup** + + ```bash + kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io + kubectl -n cpaas-system get machineinventories.elemental.cattle.io -o yaml + ``` + + The released inventory must show `baremetal.alauda.io/allocation-state=Available`, the `baremetal.alauda.io/owner-*` annotations must be cleared, and the pool annotation must remain. `MachineInventory` itself is **not** deleted. + +### Replacing a Single Failed Node \{#replacing-a-node} + +If a `BaremetalMachine` lands in `Failed`, the safest recovery is to delete the failed `Machine`. CAPI immediately creates a replacement `Machine` (because `replicas` did not change), and the bare-metal provider picks an `Available` inventory from the same pool. Most often the replacement is a different `MachineInventory`; the provider does not guarantee that the freshly released inventory will be re-picked. + +```bash +kubectl -n cpaas-system delete machine +kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io -w +``` + +Investigate the original failure separately: read `BaremetalMachine.status.conditions`, `MachineInventory.status.plan.state`, and the failing plan secret's `failed-output` key on the host that ran the plan. Common root causes are documented in the [Common Failure Modes](#common-failure-modes) section below. + +### Upgrading Machine Infrastructure + +`BaremetalMachineTemplate` carries only the pool reference and allocation policy β€” there is no CPU / memory / disk spec to revisit. Infrastructure-side changes that require a template swap are limited to: + +- Moving a `MachineDeployment` to a different pool. +- Adjusting allocation policy (when more policies are added). + +To swap templates safely: + +1. Create a new `BaremetalMachineTemplate` with a new `metadata.name` referencing the new pool. +2. Apply the new template. +3. Patch the `MachineDeployment`: + + ```bash + kubectl -n cpaas-system patch machinedeployment -workers \ + --type='merge' \ + -p='{"spec":{"template":{"spec":{"infrastructureRef":{"name":""}}}}}' + ``` + +4. Watch the rolling replacement complete (`maxSurge=0`, one node at a time). + +### Updating Bootstrap Templates \{#updating-bootstrap-templates} + +`KubeadmConfigTemplate` is an immutable template in the same sense as `BaremetalMachineTemplate`. Modifying an existing template in place does not roll out existing machines; only newly created machines pick up the changes. + +To roll out a bootstrap change: + +1. Export the existing template: + + ```bash + kubectl -n cpaas-system get kubeadmconfigtemplate -worker-bootstrap -o yaml \ + > new-worker-bootstrap.yaml + ``` + +2. Change `metadata.name`, remove server-generated fields (`resourceVersion`, `uid`, `creationTimestamp`, `managedFields`, `kubectl.kubernetes.io/last-applied-configuration`) and the entire `status`, and edit the desired fields. +3. Apply the new template: + + ```bash + kubectl apply -f new-worker-bootstrap.yaml + ``` + +4. Patch the `MachineDeployment` to reference the new template: + + ```bash + kubectl -n cpaas-system patch machinedeployment -workers \ + --type='merge' \ + -p='{"spec":{"template":{"spec":{"bootstrap":{"configRef":{"name":""}}}}}}' + ``` + + This triggers a rolling replacement. + +### Upgrading Kubernetes Version + +For Kubernetes upgrades on bare-metal, see [Upgrading Clusters on Bare Metal](../upgrade-cluster/bare-metal.mdx). The upgrade path always replaces nodes β€” there is no in-place `kubeadm upgrade` step. + +--- + +## Pool and Inventory Observability + +The annotations the provider maintains on each `MachineInventory` are the authoritative source for "what is this host being used for right now": + +| Annotation | Values | Meaning | +|---|---|---| +| `baremetal.alauda.io/pool` | Pool name | Which pool owns this inventory. | +| `baremetal.alauda.io/allocation-state` | `Available` / `Allocated` / `Preparing` / `Reprovisioning` / `Unavailable` | Lifecycle phase from the provider's point of view. | +| `baremetal.alauda.io/owner-cluster` | Cluster name | Cluster currently using this inventory (cleared on release). | +| `baremetal.alauda.io/owner-machine` | `Machine` name | Owning Cluster API `Machine`. | +| `baremetal.alauda.io/owner-baremetalmachine` | `BaremetalMachine` name | Owning provider machine. | + +Each `BaremetalMachine.status.planSecretRef` plan secret also carries `baremetal.alauda.io/plan.type=clean|reprovision` so you can distinguish which plan is currently being driven. + +--- + +## Common Failure Modes \{#common-failure-modes} + +| Scenario | Expected condition | What to check | +|---|---|---| +| Pool absent | `BaremetalMachine` stays `Pending`, `InventoryAllocated=False / Reason=PoolMissing` | `BaremetalMachineTemplate.spec.template.spec.machineInventoryPoolRef.name` and namespace | +| Pool exhausted | `BaremetalMachine` stays `Pending`, `InventoryAllocated=False / Reason=PoolExhausted` | `MachineInventoryPool.status.available`; inventory ownership annotations | +| Member missing | `MachineInventoryPool.MembersValid=False / Ready=False` | Pool's `status.unavailable`, message on the failing condition | +| Inventory not Ready | Counted toward `status.unavailable` | `MachineInventory.status.conditions[Ready]`, plan secret existence | +| Plan secret missing | Inventory ineligible for allocation | `MachineInventory.status.plan.secretRef`, presence of the referenced Secret | +| Bootstrap secret missing | `BootstrapReady=False / Reason=BootstrapWaiting` | `Machine.spec.bootstrap.dataSecretName` | +| Catalog miss | `BaremetalMachine` `Failed`, `ImageResolved=False / Reason=ImageCatalogMiss`, no plan written | `elemental-image-catalog` keys | +| Registry annotation missing | `ImageResolved=False / Reason=ImageRegistryMissing` | `Cluster.metadata.annotations["cpaas.io/registry-address"]` | +| Reprovision plan failed | `BaremetalMachine` `Failed`, inventory marked `Unavailable` | Plan secret `failed-output`; host serial console; reachable platform registry | +| Clean plan failed | Deletion blocked by finalizer; inventory marked `Unavailable` | Same as above, focused on `clean` plan output | + +For the full operator-side state machine reference, see [Provider Overview β†’ BaremetalMachine](../overview/providers/bare-metal.mdx#baremetalmachine). + +--- + +## Next Steps + +- [Create Clusters](../create-cluster/bare-metal.mdx) +- [Upgrade Clusters](../upgrade-cluster/bare-metal.mdx) diff --git a/docs/en/manage-nodes/index.mdx b/docs/en/manage-nodes/index.mdx index 110988bc..90ef8586 100644 --- a/docs/en/manage-nodes/index.mdx +++ b/docs/en/manage-nodes/index.mdx @@ -22,7 +22,7 @@ Select your platform for detailed node management instructions: - [Huawei DCS](./huawei-dcs.mdx) - [Huawei Cloud Stack](./huawei-cloud-stack.mdx) - [VMware vSphere](./vmware-vsphere.mdx) -- Bare Metal (Planned) +- [Bare Metal](./bare-metal.mdx) ## Related Topics diff --git a/docs/en/overview/providers/bare-metal.mdx b/docs/en/overview/providers/bare-metal.mdx index 01fb0b89..90ff37a5 100644 --- a/docs/en/overview/providers/bare-metal.mdx +++ b/docs/en/overview/providers/bare-metal.mdx @@ -1,36 +1,182 @@ --- weight: 40 +title: Bare Metal Provider +author: dev@alauda.io +category: introduction +queries: + - bare metal provider + - baremetal cluster management + - elemental operator baremetal + - machineinventorypool --- # Bare Metal Provider -The Bare Metal Infrastructure Provider enables Immutable Infrastructure on bare metal servers without virtualization. - ## Overview -The Bare Metal Provider manages physical servers directly, without any virtualization layer. This is ideal for scenarios requiring maximum performance or when virtualization infrastructure is not available. +The Bare Metal Infrastructure Provider enables Immutable Infrastructure on physical servers, with no virtualization layer in between. It composes two long-running components on the `global` cluster: + +- **`elemental-operator`** β€” registers physical hosts, builds installation ISOs (`SeedImage`), and maintains the long-lived `MachineInventory` object for every host. `elemental-system-agent` runs on each host and executes `MachineInventory` plan secrets. +- **`cluster-api-provider-baremetal`** β€” the Cluster API infrastructure provider. It groups available `MachineInventory` objects into pools, binds each `Machine` to a `MachineInventory`, and writes `clean` / `reprovision` plans that drive the host through Kubernetes node lifecycle. + +Unlike VM providers (DCS, vSphere), the bare-metal provider does **not** create or destroy machines. A host is installed once (live ISO β†’ on-disk OS via `elemental install`), registers itself as a `MachineInventory`, and stays in the inventory across the cluster lifecycle. Node "creation" and "deletion" are realized through `elemental upgrade` driven by plans, followed by reboot and cloud-init re-execution. ## Status -πŸ“‹ **Planned** +The provider currently follows a YAML-only workflow. There is no Fleet Essentials UI for bare-metal clusters yet β€” every step on this page is driven by `kubectl apply`. + +## Key Features + +- **`MachineInventoryPool` allocation model** β€” operators pre-declare which `MachineInventory` objects may back a `KubeadmControlPlane` or `MachineDeployment`. The provider picks an `Available` inventory from that pool when a `Machine` is created; nothing is provisioned outside the declared set. +- **Plan-driven node lifecycle** β€” node attach uses a `reprovision` plan (write cloud-init, `cloud-init clean`, `elemental upgrade`, reboot, cloud-init re-execute, `kubeadm init/join`). Node detach uses a `clean` plan (stop kubelet, clear CRI workload, stop containerd). `MachineInventory` is never deleted by the provider during scaling, upgrade, or cluster deletion. +- **Cluster API native object tree** β€” uses upstream `Cluster`, `KubeadmControlPlane`, `MachineDeployment`, `Machine`. The provider only owns the infrastructure tree (`BaremetalCluster`, `BaremetalMachine`, `BaremetalMachineTemplate`, `MachineInventoryPool`). There is no custom control-plane CRD. +- **Image-catalog driven Kubernetes versions** β€” the provider keeps a cluster-scoped `elemental-image-catalog` ConfigMap that maps `Machine.spec.version` to an `elemental upgrade` image. Upgrades become "patch the version, controller resolves the matching image, reprovisions the node." +- **Control-plane HA via `alive`** β€” `BaremetalCluster.spec.controlPlaneLoadBalancer` declares a control-plane VIP and VRID; the provider deploys the `alive` chart (keepalived + IPVS + `kube-lock` Lease arbitration) onto the control-plane nodes once the workload cluster is reachable. +- **Network identity preservation** β€” `elemental-register` reports the live-ISO observed network as `MachineInventory.spec.observedNetwork`. The first `elemental install` and every later `reprovision` plan replay that snapshot as cloud-init `network-config v2`, so a host keeps its address, default route, and DNS across the entire lifecycle. + +## Differences from VM-Based Providers + +| Aspect | VM providers (DCS, vSphere) | Bare-metal provider | +|---|---|---| +| Node creation | Clone a VM template from a credential-scoped platform | Reprovision an already-installed physical host via `elemental upgrade` | +| Node deletion | Power-off and delete the VM | Run `clean` plan; host stays in inventory and returns to pool | +| Pool model | IP pool / hostname pool sized per replica | `MachineInventoryPool` listing concrete `MachineInventory` names | +| Data on the node | System and template disks recreated on every replacement; declared persistent disks survive (DCS) | `elemental upgrade` snapshot model β€” `initramfs` clears Kubernetes persistent state; pool-managed persistent disks are not part of this provider | +| Recovery time | Minutes per replacement | Minutes per `elemental upgrade` + reboot | +| First install path | One-off VM template upload by the platform admin | Live ISO built by `SeedImage`, booted on the host, `elemental install` to disk | +| Control-plane LB | External LB supplied by operator | Internal LB managed by `alive` static pods on the control-plane nodes (External LB supported through `type: External`) | +| UI support | YAML + Fleet Essentials UI (DCS `1.0.13` and later) | YAML only | + +## Concepts and Terminology \{#concepts-and-terminology} + +### Object hierarchy + +```text +elemental.cattle.io infrastructure.cluster.x-k8s.io +───────────────────── ──────────────────────────────── +MachineRegistration ──┐ BaremetalCluster + β”‚ BaremetalMachineTemplate +SeedImage ─── BaremetalMachine + β–Ό MachineInventoryPool +MachineInventory ◄──────────────────────────► (referenced by name) + β–² + β”‚ (default plan secret, status.plan) + β–Ό +elemental-system-agent on the host +``` + +`elemental-operator` owns the left column (host registration and the long-lived `MachineInventory`). `cluster-api-provider-baremetal` owns the right column (the Cluster API infrastructure tree) and only references `MachineInventory` by name. + +### Bare-metal concepts + +#### `MachineRegistration` \{#machineregistration} + +Declares the registration endpoint and first-install cloud-config that `elemental-register` consumes on the live ISO. Operators set `machineName`, `machineInventoryLabels`, and `machineInventoryAnnotations` (with `${SMBIOS/...}` templating) plus the `elemental.install` and `elemental.registration` blocks. `MachineRegistration` is queried but not modified by the bare-metal provider β€” it belongs to the elemental layer. + +#### `SeedImage` \{#seedimage} + +Triggers `elemental-operator` to build a bootable ISO that contains the registration URL and the operator's TLS material baked into it. `spec.baseImage` must reference the **ISO** variant of the OS image that matches the target Kubernetes version (the repository name carries the `-iso` suffix; the tag/digest matches the `elemental-image-catalog` entry for the version you intend to install). The ISO is booted once per physical host; on boot, the host runs `elemental-register` followed by `elemental install` and creates a `MachineInventory`. + +#### `MachineInventory` \{#machineinventory} + +The long-lived host identity object. The provider only relies on the following parts of its contract: + +- `status.plan.secretRef` β€” the single default plan secret owned by `elemental-operator`. +- `status.plan.state` β€” `Applied` / `Failed` transitions used to drive the `BaremetalMachine` state machine. +- `status.conditions` β€” host-side readiness signals. +- `spec.observedNetwork` β€” fork-only field populated by `elemental-register` from the live-ISO NICs; replayed during install and during every `reprovision` plan. + +The provider never deletes a `MachineInventory`, never uses `MachineInventorySelector`, and does not run a separate `MachineInventoryLifecycleController` β€” that lifecycle remains with `elemental-operator`. + +#### `MachineInventoryPool` \{#machineinventorypool} + +Operator-authored set of `MachineInventory` names that a given `BaremetalMachineTemplate` is allowed to draw from. Pools are scoped to a single `clusterName`, and each `MachineInventory` belongs to at most one active pool at a time. The pool reconciler aggregates the pool-wide capacity counters used everywhere in the docs: + +- `available` β€” free for allocation +- `allocated` β€” bound to an active `BaremetalMachine` +- `preparing` β€” running a `clean` plan +- `reprovisioning` β€” running a `reprovision` plan +- `unavailable` β€” Ready=False, plan failed, missing plan secret, or not present in the cluster + +#### `BaremetalCluster` \{#baremetalcluster} -This provider is in the planning phase. +The Cluster API infrastructure cluster resource. Owns `controlPlaneLoadBalancer` (the VIP and `vrid` consumed by the `alive` chart) and `controlPlaneEndpoint` (backfilled from `controlPlaneLoadBalancer` when only the VIP is set). The reconciler defers cluster-addon deployment (`alive`, `kube-ovn`) until the workload control plane is reachable. -## Key Considerations +#### `BaremetalMachineTemplate` \{#baremetalmachinetemplate} -- **In-place Upgrade**: Essential for small-scale environments (3-5 machines) where rolling upgrade is not feasible -- **Data Persistence**: Critical for bare metal scenarios since data disks cannot be detached -- **BMC Integration**: Leverage Baseboard Management Controller for remote management and self-healing +The Cluster API infrastructure template referenced by `KubeadmControlPlane.spec.machineTemplate.infrastructureRef` and by `MachineDeployment.spec.template.spec.infrastructureRef`. Templates only carry `machineInventoryPoolRef` (which pool this machine group draws from) and `allocationPolicy` (`Ordered` is currently the only supported value β€” picks the first `Available` inventory in declaration order). -## Differences from VM-based Providers +There is deliberately no `version`, `role`, or `upgradeImage` on the template. Role comes from the owning Cluster API resource; the version comes from `Machine.spec.version`; the upgrade image is resolved at reprovision time from the global image catalog. -| Feature | VM Providers (DCS, vSphere) | Bare Metal Provider | -|---------|----------------------------|---------------------| -| Upgrade Method | Rolling Replace (create new, delete old) | In-place Upgrade | -| Data Persistence | Disk detach/attach | Must persist in-place | -| Self-Healing | Delete and recreate VM | Reboot or Reprovision | -| Recovery Time | Minutes | Depends on physical provisioning | +#### `BaremetalMachine` \{#baremetalmachine} + +The Cluster API infrastructure machine. Reconciles a single `Machine` against a single `MachineInventory`: + +1. Picks an `Available` inventory from the pool referenced by the owning template. +2. Reads the owning `Machine.spec.bootstrap.dataSecretName` and resolves the elemental upgrade image for `Machine.spec.version` from the image catalog. +3. Normalizes the bootstrap user-data (hostname, kubelet `provider-id`, `criSocket`) and writes the `reprovision` plan into the `MachineInventory` plan secret. +4. Watches `MachineInventory.status.plan.state` until the plan reports `Applied`, then sets `BaremetalMachine.status.providerID = baremetal:///`. +5. On deletion, writes a `clean` plan and clears the owner annotations once the plan applies, returning the inventory to the pool. + +Phase transitions: `Pending β†’ Allocated β†’ Reprovisioning β†’ Running`; deletion: `Running β†’ Preparing β†’ Deleted`; failure: `* β†’ Failed`. + +#### Image catalog \{#image-catalog} + +A cluster-scoped `ConfigMap` (default name `elemental-image-catalog` in `cpaas-system`) that maps `Machine.spec.version` to an `elemental upgrade` image. The bare-metal provider chart renders this ConfigMap from `provider.imageCatalog.images` (`global.registry.address` is prepended to each repository) and from `provider.imageCatalog.data` (for fully-qualified overrides such as digest-pinned images). The reconciler hot-reloads the ConfigMap; a missing key is a terminal `Failed` state, not a fallback to a default image. + +The image catalog also drives `SeedImage.spec.baseImage`: the ISO variant is the same repository with `-iso` appended and the same tag/digest as the catalog entry. + +#### `clean` and `reprovision` plans \{#clean-reprovision} + +The only two plan types the provider writes into `MachineInventory.spec.plan` (annotated with `baremetal.alauda.io/plan.type=clean|reprovision`): + +- **`reprovision`** β€” runs on node attach. Writes NoCloud `user-data` / `meta-data` (and, when `MachineInventory.spec.observedNetwork` is non-empty, `network-config` v2), writes a cleanup marker, runs `cloud-init clean --logs --seed`, runs `elemental upgrade --reboot=false --system `, and triggers a delayed reboot. After reboot, `initramfs` clears `/var/lib/kubelet`, `/var/lib/containerd`, `/var/lib/etcd`, `/etc/kubernetes`; cloud-init re-runs and performs `kubeadm init` / `kubeadm join`. +- **`clean`** β€” runs on node detach. Stops `kubelet`, clears CRI workload, stops `containerd`. It explicitly **does not** run `kubeadm reset`, `cloud-init clean`, or `elemental upgrade`, and it does not reboot. Real cleanup is deferred to the `reprovision` plan that runs when the host is re-allocated. + +The provider applies the plans through the upstream "single default plan secret" semantics; it does not use `MachineInventorySelector` or `FleetBundle`. + +#### `alive` (control-plane HA) \{#alive} + +`alive` is a set of static pods (keepalived + IPVS) plus a `kube-lock` Lease arbitrator that maintain the control-plane VIP described in `BaremetalCluster.spec.controlPlaneLoadBalancer`. The provider deploys `alive` as an `AppRelease` on the workload cluster once the first control-plane Node is Ready, and re-renders it whenever the control-plane membership changes. During the very first `kubeadm init`, the provider prepends a one-off `ip addr add /32 dev eth0` command to the bootstrap so that the first node holds the VIP until `alive` takes over. + +The VIP must live in the control-plane nodes' Layer-2 broadcast domain; `vrid` must be unique within that domain. + +#### `MachineInventory.spec.observedNetwork` \{#observed-network} + +Fork-only field populated by `elemental-register` from the live-ISO NIC state and reported back via the `MsgObservedNetworkConfig` registration message. It is consumed in two places: + +- During the first `elemental install`, when `MachineInventory.spec.network` is empty, the registration server falls back to translating `observedNetwork` into an `nmconnections` `NetworkConfig` so the on-disk OS keeps the same address that the live ISO had. +- During every `reprovision`, the provider translates `observedNetwork` into cloud-init `network-config` v2 (a netplan subset) and writes it as the third NoCloud seed file. Explicit `spec.network` takes precedence in both cases. + +## API Group + +All bare-metal infrastructure resources belong to `infrastructure.cluster.x-k8s.io/v1beta1`. Elemental resources belong to `elemental.cattle.io/v1beta1`. + +| Resource | Description | Documentation | +|---|---|---| +| `BaremetalCluster` | Cluster-level infrastructure (control-plane VIP, endpoint, network type) | [BaremetalCluster](../../apis/providers/bare-metal/baremetalcluster.mdx) | +| `BaremetalMachine` | Single infra Machine bound to one `MachineInventory` | [BaremetalMachine](../../apis/providers/bare-metal/baremetalmachine.mdx) | +| `BaremetalMachineTemplate` | Template that binds a pool to a `KubeadmControlPlane` or `MachineDeployment` | [BaremetalMachineTemplate](../../apis/providers/bare-metal/baremetalmachinetemplate.mdx) | +| `MachineInventoryPool` | Allowed set of `MachineInventory` names for a cluster | [MachineInventoryPool](../../apis/providers/bare-metal/machineinventorypool.mdx) | + +## Supported Kubernetes Versions + +The bare-metal provider supports the Kubernetes versions listed in its `elemental-image-catalog`. The default chart values ship two entries (the `v1.33.7-2` and `v1.34.5` `baremetal-base-image` releases); additional versions are introduced by appending entries to `provider.imageCatalog.images` (renders to `global.registry.address/:`) or by overriding with a full image reference under `provider.imageCatalog.data`. See [OS Support Matrix](../os-support-matrix.mdx) for the matching component versions. + +## Requirements + +- Physical hosts (or PXE-bootable VMs for lab use) with BIOS or UEFI access to mount the `SeedImage` ISO. +- A platform registry reachable from each host (used by both `elemental install` and `elemental upgrade`). Set `global.registry.address` and, when the registry is self-signed, leave `global.registry.tlsVerify=false` (chart default). +- A control-plane VIP, free port (typically `6443`), and a `vrid` unique within the control-plane Layer-2 broadcast domain. +- One `MachineInventoryPool` per role (control plane, worker) sized to at least the target replica count plus headroom for upgrades. +- TPM available on production hosts. Lab and PoC hosts can keep `MachineRegistration.spec.config.elemental.registration.emulate-tpm: true` to bypass real TPM hardware. ## Documentation -Documentation will be available when the provider is released. +For detailed instructions on using the bare-metal provider, see: + +- [Installation](../../install/bare-metal.mdx) +- [Creating Clusters](../../create-cluster/bare-metal.mdx) +- [Managing Nodes](../../manage-nodes/bare-metal.mdx) +- [Upgrading Clusters](../../upgrade-cluster/bare-metal.mdx) +- [API Reference](../../apis/providers/bare-metal/) diff --git a/docs/en/overview/providers/index.mdx b/docs/en/overview/providers/index.mdx index 0a070fe0..2a7f21ca 100644 --- a/docs/en/overview/providers/index.mdx +++ b/docs/en/overview/providers/index.mdx @@ -22,7 +22,7 @@ Immutable Infrastructure supports multiple infrastructure platforms through plug | [Huawei DCS](./huawei-dcs.mdx) | Huawei Datacenter Virtualization Solution | βœ… Available | | [Huawei Cloud Stack](./huawei-cloud-stack.mdx) | Huawei Cloud Stack | βœ… Available | | [VMware vSphere](./vmware-vsphere.mdx) | VMware vSphere virtualization platform | βœ… Available | -| [Bare Metal](./bare-metal.mdx) | Bare-metal servers without virtualization | πŸ“‹ Planned | +| [Bare Metal](./bare-metal.mdx) | Bare-metal servers without virtualization | βœ… Available (YAML only) | ## Control Plane Endpoint @@ -33,8 +33,9 @@ Each cluster's Kubernetes API server (`kube-apiserver`, port `6443`) is reached | Huawei DCS | You provide it | An external load balancer that fronts the control plane nodes, or a single control plane node reached directly | Provision the load balancer before cluster creation and supply its address and port in `DCSCluster.spec.controlPlaneLoadBalancer` | | Huawei Cloud Stack | The provider creates it | An HCS Elastic Load Balance (ELB) instance created automatically during cluster reconciliation | Plan the ELB inputs β€” VIP subnet, VIP address, bandwidth, and public-IP preference β€” in `HCSCluster.spec.controlPlaneLoadBalancer` | | VMware vSphere | You provide it | An external load balancer that fronts the control plane nodes | Provision the load balancer and allocate the control plane VIP before cluster creation; the documented flow does not deploy an in-cluster VIP component | +| Bare Metal | The provider creates it (or you provide it) | `BaremetalCluster.spec.controlPlaneLoadBalancer.type=Internal` deploys the `alive` chart (`keepalived` + IPVS + `kube-lock` Lease arbitration) onto the control-plane nodes; `type=External` delegates the LB to an existing external appliance | Choose a free VIP, port (typically `6443`), and a `vrid` unique in the control-plane Layer-2 broadcast domain | -Huawei Cloud Stack is the only provider that creates the control plane load balancer for you. For Huawei DCS and VMware vSphere, the load balancer β€” including its VIP allocation and backend (real-server) maintenance β€” is prepared and owned outside the cluster. Bare Metal is planned; its control plane endpoint model will be documented when the provider is released. +Huawei Cloud Stack is the only VM provider that creates the control plane load balancer for you. For Huawei DCS and VMware vSphere, the load balancer β€” including its VIP allocation and backend (real-server) maintenance β€” is prepared and owned outside the cluster. On Bare Metal, the provider can either deploy an in-cluster VIP component (`alive`) or defer to an external LB; see the [Bare Metal Provider](./bare-metal.mdx) overview for the full model. ## Fleet Essentials Plugin @@ -53,6 +54,7 @@ Fleet Essentials works with Infrastructure Providers to deliver platform-specifi - **DCS Provider**: Alauda Container Platform DCS Infrastructure Provider `1.0.13` and later adds DCS-specific UI pages and workflows on Huawei DCS. Pool-managed persistent-disk scenarios require DCS provider `v1.0.16` or later, and in `v1.0.16` the `DCSIpHostnamePool.spec.pool[].persistentDisk` configuration remains YAML-only - **HCS Provider**: Provides YAML-based cluster lifecycle management on Huawei Cloud Stack. HCS provider `v1.0.1` and later supports pool-managed persistent disks through `HCSMachineConfigPool.spec.configs[].persistentDisks[]` - **vSphere Provider**: The VMware vSphere infrastructure provider is generally available; its Fleet Essentials UI extension for cluster creation and management is coming soon +- **Bare Metal Provider**: Provides YAML-only cluster lifecycle management on physical servers. There is no Fleet Essentials UI for bare-metal clusters; every workflow is driven by `kubectl apply` against `BaremetalCluster`, `BaremetalMachineTemplate`, `MachineInventoryPool`, and the upstream CAPI resources ## Provider Architecture diff --git a/docs/en/upgrade-cluster/bare-metal.mdx b/docs/en/upgrade-cluster/bare-metal.mdx index 27f902f2..eb6e3ec4 100644 --- a/docs/en/upgrade-cluster/bare-metal.mdx +++ b/docs/en/upgrade-cluster/bare-metal.mdx @@ -1,17 +1,214 @@ --- weight: 40 +title: Upgrading Kubernetes on Bare Metal +author: dev@alauda.io +category: howto +queries: + - upgrade kubernetes on bare metal + - baremetal kubernetes upgrade guide + - upgrade baremetal worker nodes + - elemental upgrade reprovision --- -# Upgrading Clusters on Bare Metal +# Upgrading Kubernetes on Bare Metal -This document provides instructions for upgrading Kubernetes clusters on bare-metal servers. +This guide explains how to complete Phase 2 of the upgrade workflow for clusters on bare metal. Before you upgrade Kubernetes, complete the Distribution Version upgrade described in [Upgrading Clusters](index.mdx). -## Status +:::info +**Where this page fits in the full ACP upgrade flow** -πŸ“‹ **Planned** +This page covers only the Kubernetes step of the upgrade. The full ACP upgrade flow β€” including upgrade artifact synchronization, ACP Core upgrade through CVO, Aligned plugin upgrades, and Agnostic plugin upgrades from Marketplace β€” is documented in the ACP product documentation. Complete those steps before you start the Kubernetes step on this page: -Documentation will be available when development begins. +- +- +- +- + +Use this page when the same cluster runs on a physical-host immutable operating system, because the Kubernetes step on bare-metal replaces every node from a new `elemental upgrade` image rather than upgrading binaries in place. +::: ## Key Considerations -Unlike VM-based platforms, bare-metal upgrades use **in-place upgrade** approach to avoid lengthy physical machine reinstallation. Critical data persists on local disks during the upgrade process. +Bare-metal upgrades replace every node β€” they do not run `kubeadm upgrade` on the existing OS. The mechanism is: + +1. CAPI deletes one `Machine` according to the rollout strategy. +2. The provider writes a `clean` plan to the inventory's plan secret. +3. The host stops kubelet + CRI workload + containerd, then returns to `Available` in the same pool. +4. CAPI creates a replacement `Machine`. +5. The provider picks an `Available` inventory from the same pool, resolves the new Kubernetes version against `elemental-image-catalog`, and writes a `reprovision` plan. +6. The host runs `cloud-init clean` β†’ `elemental upgrade --system ` β†’ reboot. `initramfs` clears Kubernetes persistent state; cloud-init re-executes and runs `kubeadm init` / `kubeadm join` against the new control plane. + +Two structural consequences operators must internalize before starting: + +- **Bare-metal does not preserve Kubernetes-managed disk state.** `/var/lib/kubelet`, `/var/lib/containerd`, `/var/lib/etcd`, `/etc/kubernetes` are cleared by the initramfs cleanup step of every reprovision. There is no equivalent of the DCS provider's pool-managed persistent disks today. +- **The same `MachineInventory` may not be re-picked.** The provider does not guarantee that the inventory released by a `clean` plan is the same one re-allocated to the replacement `Machine`. Replacements are performed as delete-then-add when `maxSurge=0` (no overlap between old and new node), so pool capacity sizing need only accommodate the desired replica count β€” old and new nodes do not coexist during the rollout. + +## Upgrade Sequence + +Upgrade bare-metal clusters in the following order: + +1. **(Prerequisite)** Upgrade the ACP platform on the `global` cluster. This brings the bare-metal provider, `elemental-operator`, and the related CAPI components to versions that understand the new schema. Trigger workload-cluster upgrades only after the management-side controllers have rolled out and become Ready. +2. Upgrade the Distribution Version (Aligned Extensions) on the workload cluster. See [Upgrading Distribution Version](index.mdx). +3. Ensure the target Kubernetes version is present in `elemental-image-catalog` (and the matching `-iso` repository is available for any future host re-registration). +4. Upgrade the control plane Kubernetes version (replaces all control-plane nodes one at a time). +5. Upgrade worker nodes to the target Kubernetes version (replaces all worker nodes within the `maxUnavailable` budget). + +Cluster API orchestrates the rolling replacement. + +:::warning +Skipping step 1 risks two failure modes: the old provider silently ignores new schema fields; or a controller image swap mid-rollout interrupts the plan secret state machine. Always settle the management-side upgrade before touching workload rollout. +::: + +## Prerequisites + +Before you start, ensure all of the following prerequisites are met: + +- The Distribution Version upgrade is complete. +- The control plane is reachable through the existing `:`. +- All current nodes are healthy and `Ready`. +- The target Kubernetes version is a key in `elemental-image-catalog`. If it is not, add it before starting (see [Update the Image Catalog](#update-the-image-catalog)). +- The platform registry is reachable from every host in the cluster. +- Both `KubeadmControlPlane.spec.rolloutStrategy.rollingUpdate.maxSurge = 0` and `MachineDeployment.spec.strategy.rollingUpdate.maxSurge = 0` are set β€” bare-metal does not over-provision physical hosts. +- The relevant pools have enough capacity to replace one node at a time without falling below the desired replica count. + +## Update the Image Catalog \{#update-the-image-catalog} + +`elemental-image-catalog` is the resource that introduces a Kubernetes version to the bare-metal provider. + +:::info +In most upgrades you do not edit this ConfigMap by hand. The bare-metal provider plugin re-renders `elemental-image-catalog` with the Kubernetes versions shipped by the new distribution when it is reapplied during the Distribution Version upgrade (Phase 1). Your task is normally just to **verify** that the target version is present (see the verification step below). The two options that follow are only needed when you must add an out-of-band version β€” for example a digest-pinned or test build that the plugin does not ship. +::: + +**Option A β€” Add a chart override.** Append the new version under `provider.imageCatalog.images` (uses `global.registry.address` as the registry): + +```yaml +provider: + imageCatalog: + images: + v1.33.7-2: + repository: tkestack/baremetal-base-image + tag: v0.0.0-beta-1.33.7-2 + : # for example, v1.34.5 + repository: tkestack/baremetal-base-image + tag: # for example, v0.0.0-dev.0-1.34.5 +``` + +Reapply the bare-metal provider plugin. The chart re-renders the ConfigMap; the provider's in-process watch hot-reloads the cache without restarting. + +**Option B β€” Patch the ConfigMap directly.** Useful for digest-pinned images or out-of-band test versions: + +```bash +kubectl -n cpaas-system patch configmap elemental-image-catalog \ + --type='json' \ + -p='[{"op":"add","path":"/data/","value":"/tkestack/baremetal-base-image:"}]' +``` + +In either case, verify before continuing: + +```bash +kubectl -n cpaas-system get configmap elemental-image-catalog -o yaml +``` + +The key must include the leading `v` (for example `v1.34.5`). If the key is missing when CAPI creates the replacement `Machine`, the resulting `BaremetalMachine` enters `Failed` with `ImageResolved=False / Reason=ImageCatalogMiss` and no `reprovision` plan is written β€” adding the key later restarts the reconciliation automatically. + +When the new version corresponds to a new MicroOS / base-image release, the ISO variant for that release should also be available before the upgrade. The bare-metal provider does not rebuild SeedImages automatically, but you may want to refresh `MachineRegistration` / `SeedImage` for **future** host onboarding so the new hosts come up on the new release. The ISO repository is the same as the base-image repository with `-iso` appended. + +## Upgrade the Control Plane \{#upgrade-control-plane} + +Patch `KubeadmControlPlane.spec.version` to the new Kubernetes version. Where component image tags are pinned (DNS, etcd), update them in the same edit: + +```bash +kubectl -n cpaas-system edit kubeadmcontrolplane -control-plane +``` + +- `spec.version` ← target Kubernetes version (must match an `elemental-image-catalog` key). +- `spec.kubeadmConfigSpec.clusterConfiguration.dns.imageTag` ← matching CoreDNS image tag for the new release. +- `spec.kubeadmConfigSpec.clusterConfiguration.etcd.local.imageTag` ← matching etcd image tag for the new release. + +The bare-metal provider does **not** require a new `BaremetalMachineTemplate` for a Kubernetes-only upgrade: the template only carries the pool reference, and the upgrade image is resolved from `Machine.spec.version` through the catalog. A new template is only required when you want to move the control plane to a different pool. + +Cluster API rolls control-plane nodes one at a time (because `maxSurge=0`): + +```bash +kubectl -n cpaas-system get kubeadmcontrolplane -control-plane -w +kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io +kubectl -n cpaas-system get machineinventorypools.infrastructure.cluster.x-k8s.io -control-plane-pool +kubectl get nodes -o wide # workload cluster +``` + +The expected sequence on each node: + +1. CAPI marks one old `Machine` for deletion; corresponding `BaremetalMachine` moves to `Preparing`. +2. The provider writes a `clean` plan; the inventory ends up `Available` with cleared owner annotations. +3. CAPI creates a new `Machine` (with the target version); a new `BaremetalMachine` is created. +4. The provider allocates an `Available` inventory from the control-plane pool, resolves the new image, and writes a `reprovision` plan. +5. The host runs `elemental upgrade --system `, reboots, and `kubeadm join`s the surviving control plane. `BaremetalMachine.status.phase` becomes `Running`. + +Repeat steps 1–5 until every control-plane node has been replaced. + +Throughout the rollout, `alive` continues to manage the VIP. As control-plane membership changes, the bare-metal provider re-renders the `alive` chart values (the `peer` list and `ipvs.ips`) and rolls out the static-pod manifests. + +## Upgrade Workers \{#upgrade-workers} + +Patch each `MachineDeployment.spec.template.spec.version` to the new Kubernetes version. CAPI replaces workers within the `maxUnavailable` budget β€” the bare-metal provider re-uses the same worker pool and the same image-catalog resolution logic as the control plane. + +```bash +kubectl -n cpaas-system patch machinedeployment -workers \ + --type='merge' \ + -p='{"spec":{"template":{"spec":{"version":""}}}}' +``` + +Watch: + +```bash +kubectl -n cpaas-system get machinedeployments.cluster.x-k8s.io -w +kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io +kubectl get nodes -o wide +``` + +Worker upgrades carry fewer knobs than the control plane: there is no `clusterConfiguration` block on `MachineDeployment`, so no DNS / etcd tags to update. Kubernetes component versions on worker nodes follow the new base image. + +If the new release also requires a bootstrap-template change (different cloud-init, different `kubeletExtraArgs`), follow [Updating Bootstrap Templates](../manage-nodes/bare-metal.mdx#updating-bootstrap-templates) β€” that is a separate template swap, independent of the version bump. + +## Cross-Version Upgrades + +Kubernetes only supports single-minor upgrades for the control plane (skew policy). To move from v1.32 β†’ v1.34: + +1. Add the v1.33 entry to `elemental-image-catalog` and upgrade the control plane to v1.33; wait for the rollout to complete. +2. Add the v1.34 entry; upgrade the control plane to v1.34. +3. Upgrade the worker `MachineDeployment` to v1.34. + +The bare-metal rollout strategy already serializes node replacement, so cross-version upgrades do not require additional pool capacity beyond what was already needed for a same-minor upgrade. + +## Verification + +After the rollout completes: + +```bash +kubectl -n cpaas-system get baremetalmachines.infrastructure.cluster.x-k8s.io +kubectl -n cpaas-system get machines.cluster.x-k8s.io +kubectl get nodes -o wide +``` + +All `BaremetalMachine` objects should be `Running`. Every CAPI `Machine` should report the new `spec.version`. Every workload `Node` should be `Ready` with the new `kubelet` version, and every `MachineInventory` plan secret should carry `baremetal.alauda.io/plan.type=reprovision` (the most recent plan applied). + +`MachineInventoryPool.status` should satisfy `available + allocated + preparing + reprovisioning + unavailable = total` with `preparing = reprovisioning = 0`. + +## Troubleshooting + +| Issue | What to check | +|---|---| +| Rollout stuck on the first new node | `BaremetalMachine.status.conditions[ImageResolved]` β€” confirm the new key is in `elemental-image-catalog` and that the `Cluster` carries `cpaas.io/registry-address`. | +| Host reboots into a state that never joins the cluster | `MachineInventory.status.plan.state` (`Failed` is terminal); inspect the host's serial console for `elemental upgrade` errors. Verify the platform registry is reachable from the host. | +| New control-plane node fails `kubeadm join` | Confirm the VIP is still reachable; `alive` may have lost the lock. Check `kubectl -n kube-system get lease`, `ipvsadm -Ln`, `keepalived` logs from the static pod. | +| Worker rollout stalls with `MachineDeployment.spec.strategy.rollingUpdate.maxSurge > 0` set | Bare-metal pools cannot honour over-provisioning. Set `maxSurge=0` and bump `maxUnavailable` to budget more parallel replacements within the pool. | +| Pool capacity exhausted mid-rollout | `MachineInventoryPool.status.available=0` and `allocated` still includes the not-yet-replaced nodes. Add another `MachineInventory` to the pool to free the queue, or wait for in-flight `Preparing` nodes to drain. | + +For the full operator-side state machine reference (every condition reason and recovery action), see [Provider Overview](../overview/providers/bare-metal.mdx#clean-reprovision). + +--- + +## Additional Resources + +- [Cluster API Upgrade Documentation](https://cluster-api.sigs.k8s.io/tasks/upgrading-clusters) +- [Kubernetes Version Skew Policy](https://kubernetes.io/docs/setup/release/version-skew-policy/) diff --git a/docs/en/upgrade-cluster/index.mdx b/docs/en/upgrade-cluster/index.mdx index fb61f16d..7ce6a696 100644 --- a/docs/en/upgrade-cluster/index.mdx +++ b/docs/en/upgrade-cluster/index.mdx @@ -100,6 +100,7 @@ See platform-specific guides: - [Huawei DCS](./huawei-dcs.mdx) - [Huawei Cloud Stack](./huawei-cloud-stack.mdx) - [VMware vSphere](./vmware-vsphere.mdx) +- [Bare Metal](./bare-metal.mdx) --- @@ -112,7 +113,7 @@ Select your platform for detailed upgrade instructions: | [Huawei DCS](./huawei-dcs.mdx) | Huawei Datacenter Virtualization Solution | βœ… Available | | [Huawei Cloud Stack](./huawei-cloud-stack.mdx) | Huawei Cloud Stack | βœ… Available | | [VMware vSphere](./vmware-vsphere.mdx) | VMware vSphere virtualization platform | βœ… Available | -| Bare Metal | Bare-metal servers without virtualization | πŸ“‹ Planned | +| [Bare Metal](./bare-metal.mdx) | Bare-metal servers without virtualization | βœ… Available (YAML only) | --- diff --git a/docs/shared/crds/providers/bare-metal/elemental.cattle.io_machineinventories.yaml b/docs/shared/crds/providers/bare-metal/elemental.cattle.io_machineinventories.yaml new file mode 100644 index 00000000..3085664e --- /dev/null +++ b/docs/shared/crds/providers/bare-metal/elemental.cattle.io_machineinventories.yaml @@ -0,0 +1,346 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.19.0 + name: machineinventories.elemental.cattle.io +spec: + group: elemental.cattle.io + names: + kind: MachineInventory + listKind: MachineInventoryList + plural: machineinventories + singular: machineinventory + scope: Namespaced + versions: + - name: v1beta1 + schema: + openAPIV3Schema: + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + properties: + ipAddressClaims: + additionalProperties: + description: ObjectReference contains enough information to let + you inspect or modify the referred object. + properties: + apiVersion: + description: API version of the referent. + type: string + fieldPath: + description: |- + If referring to a piece of an object instead of an entire object, this string + should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. + For example, if the object reference is to a container within a pod, this would take on a value like: + "spec.containers{name}" (where "name" refers to the name of the container that triggered + the event) or if no container name is specified "spec.containers[2]" (container with + index 2 in this pod). This syntax is chosen only to have some well-defined way of + referencing a part of an object. + type: string + kind: + description: |- + Kind of the referent. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + name: + description: |- + Name of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + type: string + namespace: + description: |- + Namespace of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ + type: string + resourceVersion: + description: |- + Specific resourceVersion to which this reference is made, if any. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency + type: string + uid: + description: |- + UID of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids + type: string + type: object + x-kubernetes-map-type: atomic + description: |- + IPAddressClaims is a map of IPAddressClaim associated to this machine. + The map key is the ipAddressPool.name. + type: object + ipAddressPools: + additionalProperties: + description: |- + TypedLocalObjectReference contains enough information to let you locate the + typed referenced object inside the same namespace. + properties: + apiGroup: + description: |- + APIGroup is the group for the resource being referenced. + If APIGroup is not specified, the specified Kind must be in the core API group. + For any other third-party types, APIGroup is required. + type: string + kind: + description: Kind is the type of resource being referenced + type: string + name: + description: Name is the name of resource being referenced + type: string + required: + - kind + - name + type: object + x-kubernetes-map-type: atomic + description: IPAddressPools is a map of IPAddressPool objects associated + to this machine. + type: object + machineHash: + description: |- + MachineHash the hash of the identifier used by the host to identify + to the operator. This is used when the host authenticates without TPM. + Both the authentication method and the identifier used to derive the hash + depend upon the MachineRegistration spec.config.elemental.registration.auth value. + type: string + network: + description: NetworkConfig is the final NetworkConfig. + properties: + config: + description: Config contains the network config template (nmc, + nmstate, or nmconnections formats) + x-kubernetes-preserve-unknown-fields: true + configurator: + default: none + description: Configurator + enum: + - none + - nmc + - nmstate + - nmconnections + type: string + ipAddresses: + additionalProperties: + type: string + description: IPAddresses contains a map of claimed IPAddresses + type: object + type: object + observedNetwork: + description: |- + ObservedNetwork is a live-ISO-side snapshot of the host's actual network + configuration uploaded by elemental-register during initial registration + (Alauda fork). It is informational only and is intentionally kept + separate from Network (the target/template network). Consumers such as + baremetal providers can read this to know what the host looked like + before install-time network reconfiguration. + properties: + dnsServers: + description: DNSServers observed (typically from /etc/resolv.conf). + items: + type: string + type: array + interfaces: + description: Interfaces observed on the host. + items: + description: ObservedInterface describes a single network interface + on the host. + properties: + addresses: + description: |- + Addresses assigned to the interface, in CIDR form (e.g. 10.0.0.5/24, + fe80::1/64). Both IPv4 and IPv6 may appear. + items: + type: string + type: array + mac: + description: MAC is the hardware address. + type: string + mtu: + description: MTU in bytes. + type: integer + name: + description: Name is the OS-level interface name (e.g. eth0, + ens3). + type: string + required: + - name + type: object + type: array + routes: + description: Routes observed in the host's routing table. + items: + description: ObservedRoute describes a single routing table + entry. + properties: + destination: + description: |- + Destination CIDR, or the literal string "default" for the default + route. + type: string + gateway: + description: Gateway is the next-hop IP, if any. + type: string + interface: + description: Interface is the outgoing device name, if any. + type: string + metric: + description: Metric of the route. + type: integer + required: + - destination + type: object + type: array + searchDomains: + description: SearchDomains observed. + items: + type: string + type: array + type: object + tpmHash: + description: |- + TPMHash the hash of the TPM EK public key. This is used if you are + using TPM2 to identify nodes. You can obtain the TPM by + running `rancherd get-tpm-hash` on the node. Or nodes can + report their TPM hash by using the MachineRegister. + type: string + type: object + status: + properties: + conditions: + description: Conditions describe the state of the machine inventory + object. + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + plan: + description: PlanStatus reflect the status of the plan owned by the + machine inventory object. + properties: + checksum: + description: Checksum checksum of the created plan. + type: string + secretRef: + description: PlanSecretRef a reference to the created plan secret. + properties: + apiVersion: + description: API version of the referent. + type: string + fieldPath: + description: |- + If referring to a piece of an object instead of an entire object, this string + should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. + For example, if the object reference is to a container within a pod, this would take on a value like: + "spec.containers{name}" (where "name" refers to the name of the container that triggered + the event) or if no container name is specified "spec.containers[2]" (container with + index 2 in this pod). This syntax is chosen only to have some well-defined way of + referencing a part of an object. + type: string + kind: + description: |- + Kind of the referent. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + name: + description: |- + Name of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + type: string + namespace: + description: |- + Namespace of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ + type: string + resourceVersion: + description: |- + Specific resourceVersion to which this reference is made, if any. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency + type: string + uid: + description: |- + UID of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids + type: string + type: object + x-kubernetes-map-type: atomic + state: + description: State reflect state of the plan that belongs to the + machine inventory. + enum: + - Applied + - Failed + type: string + type: object + type: object + type: object + served: true + storage: true + subresources: + status: {} diff --git a/docs/shared/crds/providers/bare-metal/elemental.cattle.io_machineregistrations.yaml b/docs/shared/crds/providers/bare-metal/elemental.cattle.io_machineregistrations.yaml new file mode 100644 index 00000000..80c32ea8 --- /dev/null +++ b/docs/shared/crds/providers/bare-metal/elemental.cattle.io_machineregistrations.yaml @@ -0,0 +1,358 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.19.0 + name: machineregistrations.elemental.cattle.io +spec: + group: elemental.cattle.io + names: + kind: MachineRegistration + listKind: MachineRegistrationList + plural: machineregistrations + singular: machineregistration + scope: Namespaced + versions: + - name: v1beta1 + schema: + openAPIV3Schema: + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + properties: + config: + description: Config the cloud config that will be used to provision + the node. + properties: + cloud-config: + x-kubernetes-preserve-unknown-fields: true + elemental: + properties: + install: + properties: + config-dir: + type: string + config-urls: + items: + type: string + type: array + debug: + type: boolean + device: + type: string + device-selector: + items: + properties: + key: + enum: + - Name + - Size + type: string + operator: + enum: + - In + - NotIn + - Gt + - Lt + type: string + values: + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + disable-boot-entry: + type: boolean + eject-cd: + type: boolean + firmware: + type: string + iso: + type: string + no-format: + type: boolean + poweroff: + type: boolean + reboot: + type: boolean + snapshotter: + default: + maxSnaps: 2 + type: loopdevice + properties: + maxSnaps: + default: 2 + description: MaxSnaps sets the maximum amount of snapshots + to keep + minimum: 2 + type: integer + type: + default: loopdevice + description: Type sets the snapshotter type for a + new installation, available options are 'loopdevice' + and 'btrfs' + type: string + type: object + system-uri: + type: string + tty: + type: string + type: object + registration: + properties: + auth: + default: tpm + type: string + ca-cert: + type: string + emulate-tpm: + type: boolean + emulated-tpm-seed: + format: int64 + type: integer + no-smbios: + type: boolean + no-toolkit: + type: boolean + url: + type: string + type: object + reset: + default: + reboot: true + reset-oem: true + reset-persistent: true + properties: + config-urls: + items: + type: string + type: array + debug: + type: boolean + disable-boot-entry: + type: boolean + enabled: + type: boolean + poweroff: + type: boolean + reboot: + default: true + type: boolean + reset-oem: + default: true + type: boolean + reset-persistent: + default: true + type: boolean + system-uri: + type: string + type: object + system-agent: + properties: + secret-name: + type: string + secret-namespace: + type: string + strictTLSMode: + type: boolean + token: + type: string + url: + type: string + type: object + type: object + network: + description: NetworkTemplate contains a map of IPAddressPools + and a schemaless network config template. + properties: + config: + description: Config contains the network config template (nmc, + nmstate, or nmconnections formats) + x-kubernetes-preserve-unknown-fields: true + configurator: + default: none + description: Configurator + enum: + - none + - nmc + - nmstate + - nmconnections + type: string + ipAddresses: + additionalProperties: + description: |- + TypedLocalObjectReference contains enough information to let you locate the + typed referenced object inside the same namespace. + properties: + apiGroup: + description: |- + APIGroup is the group for the resource being referenced. + If APIGroup is not specified, the specified Kind must be in the core API group. + For any other third-party types, APIGroup is required. + type: string + kind: + description: Kind is the type of resource being referenced + type: string + name: + description: Name is the name of resource being referenced + type: string + required: + - kind + - name + type: object + x-kubernetes-map-type: atomic + description: IPAddresses contains a map of IPPools references + type: object + type: object + type: object + machineInventoryAnnotations: + additionalProperties: + type: string + description: MachineInventoryAnnotations annotations to be added to + the created MachineInventory object. + type: object + machineInventoryLabels: + additionalProperties: + type: string + description: MachineInventoryLabels label to be added to the created + MachineInventory object. + type: object + machineName: + type: string + type: object + status: + properties: + conditions: + description: Conditions describe the state of the machine registration + object. + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + registrationToken: + description: RegistrationToken a token for registering a machine. + type: string + registrationURL: + description: RegistrationURL is the URL for registering a new machine. + type: string + serviceAccountRef: + description: ServiceAccountRef a reference to the service account + created by the machine registration. + properties: + apiVersion: + description: API version of the referent. + type: string + fieldPath: + description: |- + If referring to a piece of an object instead of an entire object, this string + should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. + For example, if the object reference is to a container within a pod, this would take on a value like: + "spec.containers{name}" (where "name" refers to the name of the container that triggered + the event) or if no container name is specified "spec.containers[2]" (container with + index 2 in this pod). This syntax is chosen only to have some well-defined way of + referencing a part of an object. + type: string + kind: + description: |- + Kind of the referent. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + name: + description: |- + Name of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + type: string + namespace: + description: |- + Namespace of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ + type: string + resourceVersion: + description: |- + Specific resourceVersion to which this reference is made, if any. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency + type: string + uid: + description: |- + UID of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids + type: string + type: object + x-kubernetes-map-type: atomic + type: object + type: object + served: true + storage: true + subresources: + status: {} diff --git a/docs/shared/crds/providers/bare-metal/elemental.cattle.io_seedimages.yaml b/docs/shared/crds/providers/bare-metal/elemental.cattle.io_seedimages.yaml new file mode 100644 index 00000000..80f987ad --- /dev/null +++ b/docs/shared/crds/providers/bare-metal/elemental.cattle.io_seedimages.yaml @@ -0,0 +1,243 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.19.0 + name: seedimages.elemental.cattle.io +spec: + group: elemental.cattle.io + names: + kind: SeedImage + listKind: SeedImageList + plural: seedimages + singular: seedimage + scope: Namespaced + versions: + - name: v1beta1 + schema: + openAPIV3Schema: + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + properties: + baseImage: + description: BaseImg the base elemental image used to build the seed + image. + type: string + buildContainer: + description: |- + BuildContainer settings for a custom container used to generate the + downloadable image. + properties: + args: + description: Args same as corev1.Container.Args + items: + type: string + type: array + command: + description: Command same as corev1.Container.Command + items: + type: string + type: array + image: + description: Image container image to run + type: string + imagePullPolicy: + description: Args same as corev1.Container.ImagePullPolicy + type: string + name: + description: Name of the spawned container + type: string + type: object + cleanupAfterMinutes: + default: 60 + description: |- + LifetimeMinutes the time at which the built seed image will be cleaned up. + If when the lifetime elapses the built image is being downloaded, the active + download will be completed before removing the built image. + Default is 60 minutes, set to 0 to disable. + format: int32 + type: integer + cloud-config: + description: CloudConfig contains cloud-config data to be put in the + generated iso. + x-kubernetes-preserve-unknown-fields: true + registrationRef: + description: MachineRegistrationRef a reference to the related MachineRegistration. + properties: + apiVersion: + description: API version of the referent. + type: string + fieldPath: + description: |- + If referring to a piece of an object instead of an entire object, this string + should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. + For example, if the object reference is to a container within a pod, this would take on a value like: + "spec.containers{name}" (where "name" refers to the name of the container that triggered + the event) or if no container name is specified "spec.containers[2]" (container with + index 2 in this pod). This syntax is chosen only to have some well-defined way of + referencing a part of an object. + type: string + kind: + description: |- + Kind of the referent. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + name: + description: |- + Name of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names + type: string + namespace: + description: |- + Namespace of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/ + type: string + resourceVersion: + description: |- + Specific resourceVersion to which this reference is made, if any. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency + type: string + uid: + description: |- + UID of the referent. + More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids + type: string + type: object + x-kubernetes-map-type: atomic + retriggerBuild: + description: RetriggerBuild triggers to build again a cleaned up seed + image. + type: boolean + size: + anyOf: + - type: integer + - type: string + default: 6442450944 + description: |- + Size specifies the size of the volume used to store the image. + Defaults to 6Gi + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + targetPlatform: + description: 'Platform specifies the target platform for the built + image. Example: linux/amd64' + example: linux/amd64 + pattern: ^$|^\S+\/\S+$ + type: string + type: + default: iso + description: |- + Type specifies the type of seed image to built. + Valid values are iso|raw + Defaults to "iso" + enum: + - iso + - raw + type: string + required: + - registrationRef + - type + type: object + status: + properties: + checksumURL: + description: ChecksumURL the URL from which the SeedImage checksum + can be downloaded once the image is built. + type: string + conditions: + description: Conditions describe the state of the seedimage object. + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + downloadToken: + description: DownloadToken a token to identify the seed image to download. + type: string + downloadURL: + description: DownloadURL the URL from which the SeedImage can be downloaded + once built. + type: string + state: + description: State reflect the state of the seed image build process. + enum: + - Initialized + - Started + - Completed + - Failed + - NotStarted + type: string + type: object + type: object + served: true + storage: true + subresources: + status: {} diff --git a/docs/shared/crds/providers/bare-metal/infrastructure.cluster.x-k8s.io_baremetalclusters.yaml b/docs/shared/crds/providers/bare-metal/infrastructure.cluster.x-k8s.io_baremetalclusters.yaml new file mode 100644 index 00000000..c0ac93b3 --- /dev/null +++ b/docs/shared/crds/providers/bare-metal/infrastructure.cluster.x-k8s.io_baremetalclusters.yaml @@ -0,0 +1,195 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.16.5 + labels: + cluster.x-k8s.io/v1beta1: v1beta1 + name: baremetalclusters.infrastructure.cluster.x-k8s.io +spec: + group: infrastructure.cluster.x-k8s.io + names: + categories: + - cluster-api + kind: BaremetalCluster + listKind: BaremetalClusterList + plural: baremetalclusters + shortNames: + - bmc + singular: baremetalcluster + scope: Namespaced + versions: + - additionalPrinterColumns: + - jsonPath: .spec.controlPlaneEndpoint.host + name: Endpoint + type: string + - jsonPath: .status.ready + name: Ready + type: boolean + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1beta1 + schema: + openAPIV3Schema: + description: BaremetalCluster is the Schema for the baremetalclusters API. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: |- + BaremetalClusterSpec describes the cluster-level infrastructure configuration + owned by the baremetal provider. See design-doc/api/cr-reference.md Β§3. + properties: + controlPlaneEndpoint: + description: |- + ControlPlaneEndpoint is the apiserver endpoint exposed to the workload plane. + CAPI contract: once set, must not change. + properties: + host: + minLength: 1 + type: string + port: + format: int32 + maximum: 65535 + minimum: 1 + type: integer + required: + - host + - port + type: object + controlPlaneLoadBalancer: + description: |- + ControlPlaneLoadBalancer describes the VIP / LB fronting the control plane. + When set, the reconciler backfills ControlPlaneEndpoint from this value. + properties: + host: + minLength: 1 + type: string + port: + format: int32 + maximum: 65535 + minimum: 1 + type: integer + type: + default: Internal + enum: + - Internal + - External + type: string + vrid: + description: VRID is the keepalived/vrrp identifier used by the + internal load balancer. + format: int32 + maximum: 255 + minimum: 0 + type: integer + required: + - host + - port + type: object + networkType: + description: |- + NetworkType identifies the CNI implementation the provider should expect. + Phase 1 treats this as metadata only. + type: string + type: object + status: + description: BaremetalClusterStatus aggregates infra-level readiness for + CAPI. + properties: + conditions: + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + failureDomains: + additionalProperties: + description: FailureDomainSpec matches the shape of sigs.k8s.io/cluster-api + v1beta1. + properties: + attributes: + additionalProperties: + type: string + type: object + controlPlane: + type: boolean + type: object + description: FailureDomains is reserved for future Phase 3 work. + type: object + ready: + description: Ready follows the CAPI infrastructure contract (required). + type: boolean + type: object + type: object + served: true + storage: true + subresources: + status: {} diff --git a/docs/shared/crds/providers/bare-metal/infrastructure.cluster.x-k8s.io_baremetalmachines.yaml b/docs/shared/crds/providers/bare-metal/infrastructure.cluster.x-k8s.io_baremetalmachines.yaml new file mode 100644 index 00000000..58638753 --- /dev/null +++ b/docs/shared/crds/providers/bare-metal/infrastructure.cluster.x-k8s.io_baremetalmachines.yaml @@ -0,0 +1,225 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.16.5 + labels: + cluster.x-k8s.io/v1beta1: v1beta1 + name: baremetalmachines.infrastructure.cluster.x-k8s.io +spec: + group: infrastructure.cluster.x-k8s.io + names: + categories: + - cluster-api + kind: BaremetalMachine + listKind: BaremetalMachineList + plural: baremetalmachines + shortNames: + - bmm + singular: baremetalmachine + scope: Namespaced + versions: + - additionalPrinterColumns: + - jsonPath: .spec.machineInventoryPoolRef.name + name: Pool + type: string + - jsonPath: .status.phase + name: Phase + type: string + - jsonPath: .status.machineInventoryRef.name + name: Inventory + type: string + - jsonPath: .status.ready + name: Ready + type: boolean + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1beta1 + schema: + openAPIV3Schema: + description: BaremetalMachine is the Schema for the baremetalmachines API. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: |- + BaremetalMachineSpec follows design-doc/api/resource-model.md Β§8. + Role / version / upgrade image deliberately absent: role is inferred from + the owning KubeadmControlPlane / MachineDeployment, version comes from the + owner Machine, and the elemental upgrade image is resolved at reprovision + time via the provider image catalog. + properties: + machineInventoryPoolRef: + description: |- + MachineInventoryPoolRef pins which pool this machine draws from. + The field is immutable once set (see webhook). + properties: + name: + minLength: 1 + type: string + required: + - name + type: object + providerID: + description: |- + ProviderID is the CAPI contract field populated once the node is + bound to a MachineInventory. + Format: baremetal:/// + type: string + required: + - machineInventoryPoolRef + type: object + status: + description: |- + BaremetalMachineStatus tracks provider-internal allocation and the reprovision + pipeline. + properties: + addresses: + description: 'Addresses surfaces addresses reported by the MachineInventory + (Phase 1: unused).' + items: + description: NodeAddress contains information for the node's address. + properties: + address: + description: The node address. + type: string + type: + description: Node address type, one of Hostname, ExternalIP + or InternalIP. + type: string + required: + - address + - type + type: object + type: array + conditions: + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + failureMessage: + description: FailureMessage is a CAPI-contract human-readable message. + type: string + failureReason: + description: FailureReason is a CAPI-contract terse reason for a terminal + failure. + type: string + lastAppliedBootstrapHash: + description: |- + LastAppliedBootstrapHash tracks the sha256 of the bootstrap user-data that + was last rendered into a reprovision plan. It prevents redundant patches. + type: string + lastAppliedPlanChecksum: + description: |- + LastAppliedPlanChecksum tracks the sha256 of the last plan body we wrote, + used to detect Applied/Failed transitions from MachineInventory.status.plan. + type: string + machineInventoryRef: + description: MachineInventoryRef is the inventory selected from the + pool. + properties: + name: + minLength: 1 + type: string + required: + - name + type: object + phase: + description: Phase is the coarse state; see BaremetalMachinePhase + constants. + enum: + - Pending + - Allocated + - Reprovisioning + - Running + - Preparing + - Failed + type: string + planSecretRef: + description: PlanSecretRef is the elemental plan secret currently + driven by the provider. + properties: + name: + minLength: 1 + type: string + required: + - name + type: object + ready: + description: Ready indicates CAPI-contract readiness (node joined + and healthy). + type: boolean + type: object + type: object + served: true + storage: true + subresources: + status: {} diff --git a/docs/shared/crds/providers/bare-metal/infrastructure.cluster.x-k8s.io_baremetalmachinetemplates.yaml b/docs/shared/crds/providers/bare-metal/infrastructure.cluster.x-k8s.io_baremetalmachinetemplates.yaml new file mode 100644 index 00000000..8532d734 --- /dev/null +++ b/docs/shared/crds/providers/bare-metal/infrastructure.cluster.x-k8s.io_baremetalmachinetemplates.yaml @@ -0,0 +1,140 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.16.5 + labels: + cluster.x-k8s.io/v1beta1: v1beta1 + name: baremetalmachinetemplates.infrastructure.cluster.x-k8s.io +spec: + group: infrastructure.cluster.x-k8s.io + names: + categories: + - cluster-api + kind: BaremetalMachineTemplate + listKind: BaremetalMachineTemplateList + plural: baremetalmachinetemplates + shortNames: + - bmmt + singular: baremetalmachinetemplate + scope: Namespaced + versions: + - additionalPrinterColumns: + - jsonPath: .spec.template.spec.machineInventoryPoolRef.name + name: Pool + type: string + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1beta1 + schema: + openAPIV3Schema: + description: BaremetalMachineTemplate lets KubeadmControlPlane and MachineDeployment + bind a pool. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: BaremetalMachineTemplateSpec is the template for BaremetalMachine. + properties: + template: + description: BaremetalMachineTemplateResource is the templated spec + embedded in a template. + properties: + metadata: + description: |- + ObjectMeta is metadata that all persisted resources must have, which includes all objects + users must create. This is a copy of customizable fields from metav1.ObjectMeta. + + ObjectMeta is embedded in `Machine.Spec`, `MachineDeployment.Template` and `MachineSet.Template`, + which are not top-level Kubernetes objects. Given that metav1.ObjectMeta has lots of special cases + and read-only fields which end up in the generated CRD validation, having it as a subset simplifies + the API and some issues that can impact user experience. + + During the [upgrade to controller-tools@v2](https://github.com/kubernetes-sigs/cluster-api/pull/1054) + for v1alpha2, we noticed a failure would occur running Cluster API test suite against the new CRDs, + specifically `spec.metadata.creationTimestamp in body must be of type string: "null"`. + The investigation showed that `controller-tools@v2` behaves differently than its previous version + when handling types from [metav1](k8s.io/apimachinery/pkg/apis/meta/v1) package. + + In more details, we found that embedded (non-top level) types that embedded `metav1.ObjectMeta` + had validation properties, including for `creationTimestamp` (metav1.Time). + The `metav1.Time` type specifies a custom json marshaller that, when IsZero() is true, returns `null` + which breaks validation because the field isn't marked as nullable. + + In future versions, controller-tools@v2 might allow overriding the type and validation for embedded + types. When that happens, this hack should be revisited. + properties: + annotations: + additionalProperties: + type: string + description: |- + annotations is an unstructured key value map stored with a resource that may be + set by external tools to store and retrieve arbitrary metadata. They are not + queryable and should be preserved when modifying objects. + More info: http://kubernetes.io/docs/user-guide/annotations + type: object + labels: + additionalProperties: + type: string + description: |- + labels is a map of string keys and values that can be used to organize and categorize + (scope and select) objects. May match selectors of replication controllers + and services. + More info: http://kubernetes.io/docs/user-guide/labels + type: object + type: object + spec: + description: |- + BaremetalMachineSpec follows design-doc/api/resource-model.md Β§8. + Role / version / upgrade image deliberately absent: role is inferred from + the owning KubeadmControlPlane / MachineDeployment, version comes from the + owner Machine, and the elemental upgrade image is resolved at reprovision + time via the provider image catalog. + properties: + machineInventoryPoolRef: + description: |- + MachineInventoryPoolRef pins which pool this machine draws from. + The field is immutable once set (see webhook). + properties: + name: + minLength: 1 + type: string + required: + - name + type: object + providerID: + description: |- + ProviderID is the CAPI contract field populated once the node is + bound to a MachineInventory. + Format: baremetal:/// + type: string + required: + - machineInventoryPoolRef + type: object + required: + - spec + type: object + required: + - template + type: object + type: object + served: true + storage: true + subresources: {} diff --git a/docs/shared/crds/providers/bare-metal/infrastructure.cluster.x-k8s.io_machineinventorypools.yaml b/docs/shared/crds/providers/bare-metal/infrastructure.cluster.x-k8s.io_machineinventorypools.yaml new file mode 100644 index 00000000..c3ec2240 --- /dev/null +++ b/docs/shared/crds/providers/bare-metal/infrastructure.cluster.x-k8s.io_machineinventorypools.yaml @@ -0,0 +1,183 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.16.5 + labels: + cluster.x-k8s.io/v1beta1: v1beta1 + name: machineinventorypools.infrastructure.cluster.x-k8s.io +spec: + group: infrastructure.cluster.x-k8s.io + names: + categories: + - cluster-api + kind: MachineInventoryPool + listKind: MachineInventoryPoolList + plural: machineinventorypools + shortNames: + - mip + singular: machineinventorypool + scope: Namespaced + versions: + - additionalPrinterColumns: + - jsonPath: .spec.clusterName + name: Cluster + type: string + - jsonPath: .status.total + name: Total + type: integer + - jsonPath: .status.available + name: Available + type: integer + - jsonPath: .status.allocated + name: Allocated + type: integer + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + name: v1beta1 + schema: + openAPIV3Schema: + description: MachineInventoryPool is the Schema for the machineinventorypools + API. + properties: + apiVersion: + description: |- + APIVersion defines the versioned schema of this representation of an object. + Servers should convert recognized schemas to the latest internal value, and + may reject unrecognized values. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources + type: string + kind: + description: |- + Kind is a string value representing the REST resource this object represents. + Servers may infer this from the endpoint the client submits requests to. + Cannot be updated. + In CamelCase. + More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds + type: string + metadata: + type: object + spec: + description: |- + MachineInventoryPoolSpec declares which MachineInventory objects may back a + particular KubeadmControlPlane / MachineDeployment. See design.md Β§β€œMachineInventoryPool”. + properties: + clusterName: + description: |- + ClusterName binds this pool to a specific Cluster. A MachineInventory + referenced here must not belong to another active pool. + minLength: 1 + type: string + machineInventories: + description: |- + MachineInventories enumerates the MachineInventory names allowed to back + BaremetalMachines that reference this pool. + items: + description: |- + PoolInventoryRef references a MachineInventory and optionally overrides the + hostname used when reprovisioning it. + properties: + hostname: + description: |- + Hostname is used as the node hostname when this inventory is selected. + When empty, the MachineInventory name is used. + type: string + name: + minLength: 1 + type: string + required: + - name + type: object + minItems: 0 + type: array + required: + - clusterName + - machineInventories + type: object + status: + description: |- + MachineInventoryPoolStatus summarises pool capacity from the provider's + allocation annotations plus the upstream MachineInventory.status.plan. + properties: + allocated: + format: int32 + type: integer + available: + format: int32 + type: integer + conditions: + items: + description: Condition contains details for one aspect of the current + state of this API Resource. + properties: + lastTransitionTime: + description: |- + lastTransitionTime is the last time the condition transitioned from one status to another. + This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable. + format: date-time + type: string + message: + description: |- + message is a human readable message indicating details about the transition. + This may be an empty string. + maxLength: 32768 + type: string + observedGeneration: + description: |- + observedGeneration represents the .metadata.generation that the condition was set based upon. + For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date + with respect to the current state of the instance. + format: int64 + minimum: 0 + type: integer + reason: + description: |- + reason contains a programmatic identifier indicating the reason for the condition's last transition. + Producers of specific condition types may define expected values and meanings for this field, + and whether the values are considered a guaranteed API. + The value should be a CamelCase string. + This field may not be empty. + maxLength: 1024 + minLength: 1 + pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$ + type: string + status: + description: status of the condition, one of True, False, Unknown. + enum: + - "True" + - "False" + - Unknown + type: string + type: + description: type of condition in CamelCase or in foo.example.com/CamelCase. + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$ + type: string + required: + - lastTransitionTime + - message + - reason + - status + - type + type: object + type: array + preparing: + format: int32 + type: integer + reprovisioning: + format: int32 + type: integer + total: + format: int32 + type: integer + unavailable: + format: int32 + type: integer + type: object + type: object + served: true + storage: true + subresources: + status: {}