From 78cc150e600623431bdde2bde6f567445ca3d5ce Mon Sep 17 00:00:00 2001 From: Manohar Reddy Date: Mon, 3 Aug 2026 10:25:03 +0200 Subject: [PATCH 1/4] Draft: document proposed automatic volume placement (host-id, pod-affinity, disable-smart-placement) Adds a new page covering the PVC annotations from the "Volume Placement at Creation Time" design doc that aren't documented anywhere yet: simplyblock.io/host-id (Tier 0 pin, plus its deprecated simplybk/ prefix), simplyblock.io/pod-affinity (Tier 1 co-location, marked Planned pending implementation), and simplyblock.io/disable-smart-placement (opt-out). Cross-links added from node-affinity.md and volume-migration.md. Co-Authored-By: Claude Sonnet 5 --- docs/maintenance-operations/node-affinity.md | 5 + .../volume-migration.md | 4 + .../usage/simplyblock-csi/volume-placement.md | 141 ++++++++++++++++++ 3 files changed, 150 insertions(+) create mode 100644 docs/usage/simplyblock-csi/volume-placement.md diff --git a/docs/maintenance-operations/node-affinity.md b/docs/maintenance-operations/node-affinity.md index 9c7b993e..719194b6 100644 --- a/docs/maintenance-operations/node-affinity.md +++ b/docs/maintenance-operations/node-affinity.md @@ -59,3 +59,8 @@ The storage node UUID (or host id) can be found using the `{{ cliname }} storage ```bash title="List all storage nodes in a storage cluster" {{ cliname }} storage-node list --cluster-id= ``` + +!!! tip + On Kubernetes, a volume can also be pinned per-PVC via the `simplyblock.io/host-id` annotation, and + (once available) automatically co-located with its consuming Pod — see + [Automatic Volume Placement](../usage/simplyblock-csi/volume-placement.md). diff --git a/docs/maintenance-operations/volume-migration.md b/docs/maintenance-operations/volume-migration.md index 4b080f53..6ecca7c9 100644 --- a/docs/maintenance-operations/volume-migration.md +++ b/docs/maintenance-operations/volume-migration.md @@ -188,6 +188,10 @@ spec: auto-rebalancing. A one-shot placement hint from initial provisioning does not pin a volume. Such volumes remain eligible for rebalancing. +!!! note + Auto-rebalancing's node-hotness signal is also planned to gate creation-time placement for new + volumes — see [Automatic Volume Placement](../usage/simplyblock-csi/volume-placement.md). + ## Volume Migration During Node Draining and Removal When a storage node is removed, the operator evacuates its volumes onto the remaining nodes before the node diff --git a/docs/usage/simplyblock-csi/volume-placement.md b/docs/usage/simplyblock-csi/volume-placement.md new file mode 100644 index 00000000..7d84ce77 --- /dev/null +++ b/docs/usage/simplyblock-csi/volume-placement.md @@ -0,0 +1,141 @@ +--- +title: "Automatic Volume Placement" +description: "How Simplyblock chooses a new volume's primary storage node — an explicit pin, Pod locality, and load-aware selection, in that order — and how to control it per PVC." +weight: 40050 +--- + +When a new logical volume is created, Simplyblock decides which storage node becomes its **primary** node +by evaluating a small set of signals, in a fixed order. The first signal that has an answer wins: + +| Tier | Signal | Where it's resolved | +|------|--------|----------------------| +| 0 — Explicit pin | `simplyblock.io/host-id` PVC annotation already set | `spdk-csi`, at volume creation | +| 1 — Pod locality | The consuming Pod was scheduled onto a worker that also runs a storage node | `spdk-csi`, at volume creation | +| 2 — Load-aware | No pin, no locality signal → least-loaded eligible node | Operator admission webhook | +| 3 — Default | None of the above fired | The storage cluster's built-in weighted-random pick | + +Tiers 0–2 all work by setting the same `simplyblock.io/host-id` annotation on the PVC before the volume is +actually created; whichever tier sets it first wins, and nothing below it overrides it. If none of them set +it, the cluster creates the volume using its ordinary default placement (Tier 3) — this is exactly what +happens today for a PVC that doesn't use any of the mechanisms below. + +This page covers Tiers 0–1 (the per-PVC signals). For Tier 2 (load-aware placement) see +[Auto-Rebalancing](../../maintenance-operations/volume-migration.md#auto-rebalancing), whose node-hotness +signal this tier reuses. + +## Tier 0: Pinning a Volume to a Specific Node + +A PVC can request a specific storage node directly, by UUID: + +```yaml title="PVC pinned to a specific storage node" +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: my-pvc + annotations: + simplyblock.io/host-id: "4e53efdd-86c9-424f-940c-e437eb6a2e95" +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + storageClassName: simplyblock-csi-sc +``` + +The value is a storage node UUID, found with `{{ cliname }} storage-node list --cluster-id=`. +This is the PVC-level equivalent of `{{ cliname }} volume add --host-id=` (see +[Node Affinity](../../maintenance-operations/node-affinity.md)) — it takes effect regardless of whether the +cluster has node affinity enabled, and always wins over Tier 1 and Tier 2. + +!!! warning "Deprecated annotation prefix" + The `simplybk/` annotation prefix (e.g. `simplybk/host-id`) is deprecated. Existing PVCs using the old + prefix continue to work for backward compatibility, but new deployments should use the + `simplyblock.io/` prefix. + +## Tier 1: Automatic Co-location with the Consuming Pod + +!!! warning "Planned" + Tier 1 is part of a proposed design and is not yet available. This section documents the intended + behavior; check the release notes before relying on it. + +On a cluster with hyper-converged or hybrid nodes, a volume performs best when its primary storage node is +the same worker node the consuming Pod actually lands on — no network hop on the data path. Tier 1 makes +this automatic: it inspects the Pod's *resolved* scheduling outcome (via standard Kubernetes CSI topology, +the same mechanism `WaitForFirstConsumer` StorageClasses already use) and, if that worker also runs a +storage node, stamps the PVC's `host-id` with it before the volume is created. + +Two things must both be true for a given PVC: + +- The storage cluster has [node affinity enabled](../../maintenance-operations/node-affinity.md) + (`--enable-node-affinity` / `StorageCluster.spec.enableNodeAffinity: true`). This is a cluster-wide + precondition — it decides whether co-location is possible at all. +- The PVC carries the opt-in annotation: + + ```yaml title="Requesting co-location with the consuming Pod" + kind: PersistentVolumeClaim + apiVersion: v1 + metadata: + name: my-pvc + annotations: + simplyblock.io/pod-affinity: "true" + spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + storageClassName: simplyblock-csi-sc + ``` + +Without this annotation, a PVC on a node-affinity-enabled cluster is *not* automatically co-located — it +falls through to Tier 2/Tier 3 like any other PVC. This lets some workloads request locality while others +on the same cluster participate in ordinary load-aware placement. + +!!! important + This requires a `WaitForFirstConsumer` StorageClass. That's the binding mode that delays provisioning + until a consumer Pod is scheduled — there's no "Pod's resolved node" to co-locate with otherwise. It + also works with any standard scheduling mechanism that ultimately pins the Pod to a node — + `nodeSelector`, node affinity, or pod affinity — but **not** with `spec.nodeName` set directly on the + Pod, since that bypasses the Kubernetes scheduler (and therefore `WaitForFirstConsumer` provisioning) + entirely. This is a general Kubernetes limitation + ([kubernetes/kubernetes#89953](https://github.com/kubernetes/kubernetes/issues/89953)), not specific to + Simplyblock. + +A worker can host more than one storage-node instance (multi-socket deployments). When more than one is +co-located on the Pod's worker, Tier 1 picks among them at random, so volumes spread across every instance +on that worker rather than piling onto one. + +## Opting Out of Automatic Placement + +!!! warning "Planned" + The opt-out annotation below applies once Tier 1 and Tier 2 both exist. Today, only Tier 0 (the + `host-id` pin) needs an explicit annotation to take effect. + +A single PVC can opt out of both automatic tiers (locality and load-aware), regardless of how the cluster +is configured: + +```yaml title="Disabling automatic placement for one PVC" +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: my-pvc + annotations: + simplyblock.io/disable-smart-placement: "true" +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + storageClassName: simplyblock-csi-sc +``` + +This is different from a `host-id` pin: a pin says "put it exactly here"; this annotation says "don't guess +for me" — the volume falls straight through to the cluster's normal default placement (Tier 3). + +## Clones and Snapshot Restores + +None of the above applies to a PVC created from a `VolumeSnapshot` or another PVC (`dataSource`) — a +clone or restore always lands on its source volume's own node, unconditionally. Annotations from this page +have no effect on those PVCs. From 84003846c7633c00c50426e70b6c1b7e699079ff Mon Sep 17 00:00:00 2001 From: Manohar Reddy Date: Mon, 3 Aug 2026 10:37:05 +0200 Subject: [PATCH 2/4] Rewrite volume placement docs to match what's actually implemented MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous draft followed the design doc's proposed mechanism, which turned out to be stale relative to the shipped code in simplyblock-operator. Corrections, verified against operator/internal/webhook/simplyblock_volume_placement_injector.go and csi-driver/pkg/spdk/controllerserver.go: - The canonical pin annotation is simplyblock.io/selected-storage-node; host-id and the legacy simplybk/host-id prefix are accepted but normalized into it automatically. Documented as deprecated, not as the primary annotation. - Load-aware placement for new volumes is real and already shipped, gated by volumeAutoPlacement.latencyBenchmarkEnabled (not .enabled, which only controls the continuous rebalancer) — it writes a non-pinning simplyblock.io/placement-hint annotation. Documented as present-tense, not "Planned". - Pod co-location (simplyblock.io/pod-affinity) has no EnableNodeAffinity precondition in the actual code — that cluster flag only affects SPDK's erasure-coding data plane, unrelated to this CSI-level mechanism. Removed the incorrect precondition from both the new page and the node-affinity.md cross-link. - simplyblock.io/disable-smart-placement is implemented and documented as-is. Also removed all "Planned" warnings now that every documented annotation and behavior is verified against the current code, and corrected the auto-rebalancing cross-reference in volume-migration.md to name latencyBenchmarkEnabled specifically. Co-Authored-By: Claude Sonnet 5 --- docs/maintenance-operations/node-affinity.md | 6 +- .../volume-migration.md | 5 +- .../usage/simplyblock-csi/volume-placement.md | 186 +++++++++--------- 3 files changed, 97 insertions(+), 100 deletions(-) diff --git a/docs/maintenance-operations/node-affinity.md b/docs/maintenance-operations/node-affinity.md index 719194b6..b996947c 100644 --- a/docs/maintenance-operations/node-affinity.md +++ b/docs/maintenance-operations/node-affinity.md @@ -61,6 +61,8 @@ The storage node UUID (or host id) can be found using the `{{ cliname }} storage ``` !!! tip - On Kubernetes, a volume can also be pinned per-PVC via the `simplyblock.io/host-id` annotation, and - (once available) automatically co-located with its consuming Pod — see + On Kubernetes, a volume's primary node can also be pinned per-PVC with the + `simplyblock.io/selected-storage-node` annotation, and a new volume can be automatically co-located + with its consuming Pod via the `simplyblock.io/pod-affinity` annotation — independent of this page's + cluster-wide `--enable-node-affinity` setting. See [Automatic Volume Placement](../usage/simplyblock-csi/volume-placement.md). diff --git a/docs/maintenance-operations/volume-migration.md b/docs/maintenance-operations/volume-migration.md index 6ecca7c9..abc102f2 100644 --- a/docs/maintenance-operations/volume-migration.md +++ b/docs/maintenance-operations/volume-migration.md @@ -189,8 +189,9 @@ spec: remain eligible for rebalancing. !!! note - Auto-rebalancing's node-hotness signal is also planned to gate creation-time placement for new - volumes — see [Automatic Volume Placement](../usage/simplyblock-csi/volume-placement.md). + Setting `latencyBenchmarkEnabled: true` also activates load-aware placement for newly created volumes, + independent of `enabled` (which only controls the continuous rebalancer above) — see + [Automatic Volume Placement](../usage/simplyblock-csi/volume-placement.md). ## Volume Migration During Node Draining and Removal diff --git a/docs/usage/simplyblock-csi/volume-placement.md b/docs/usage/simplyblock-csi/volume-placement.md index 7d84ce77..d725dd54 100644 --- a/docs/usage/simplyblock-csi/volume-placement.md +++ b/docs/usage/simplyblock-csi/volume-placement.md @@ -1,31 +1,25 @@ --- title: "Automatic Volume Placement" -description: "How Simplyblock chooses a new volume's primary storage node — an explicit pin, Pod locality, and load-aware selection, in that order — and how to control it per PVC." +description: "How Simplyblock chooses a new volume's primary storage node: an explicit pin, automatic load-aware placement, or co-location with the consuming Pod — and how to control each with PVC annotations." weight: 40050 --- -When a new logical volume is created, Simplyblock decides which storage node becomes its **primary** node -by evaluating a small set of signals, in a fixed order. The first signal that has an answer wins: +When a new PersistentVolumeClaim is provisioned, Simplyblock decides which storage node becomes the +volume's primary node. Up to three signals can apply, evaluated in this order — the first one available +wins: -| Tier | Signal | Where it's resolved | -|------|--------|----------------------| -| 0 — Explicit pin | `simplyblock.io/host-id` PVC annotation already set | `spdk-csi`, at volume creation | -| 1 — Pod locality | The consuming Pod was scheduled onto a worker that also runs a storage node | `spdk-csi`, at volume creation | -| 2 — Load-aware | No pin, no locality signal → least-loaded eligible node | Operator admission webhook | -| 3 — Default | None of the above fired | The storage cluster's built-in weighted-random pick | +| Order | Signal | Set via | Set by | +|-------|------------------------|-------------------------------------------------|-------------------------| +| 1 | Explicit pin | `simplyblock.io/selected-storage-node` | You | +| 2 | Load-aware placement | `simplyblock.io/placement-hint` | The operator, automatically | +| 3 | Pod co-location | `simplyblock.io/pod-affinity: "true"` (opt-in) | You | -Tiers 0–2 all work by setting the same `simplyblock.io/host-id` annotation on the PVC before the volume is -actually created; whichever tier sets it first wins, and nothing below it overrides it. If none of them set -it, the cluster creates the volume using its ordinary default placement (Tier 3) — this is exactly what -happens today for a PVC that doesn't use any of the mechanisms below. +A plain PVC with none of the above falls straight through to the storage cluster's own built-in default +placement, exactly as it does today. -This page covers Tiers 0–1 (the per-PVC signals). For Tier 2 (load-aware placement) see -[Auto-Rebalancing](../../maintenance-operations/volume-migration.md#auto-rebalancing), whose node-hotness -signal this tier reuses. +## Pinning a Volume to a Specific Node -## Tier 0: Pinning a Volume to a Specific Node - -A PVC can request a specific storage node directly, by UUID: +Use `simplyblock.io/selected-storage-node` to request a specific storage node UUID directly: ```yaml title="PVC pinned to a specific storage node" kind: PersistentVolumeClaim @@ -33,7 +27,7 @@ apiVersion: v1 metadata: name: my-pvc annotations: - simplyblock.io/host-id: "4e53efdd-86c9-424f-940c-e437eb6a2e95" + simplyblock.io/selected-storage-node: "4e53efdd-86c9-424f-940c-e437eb6a2e95" spec: accessModes: - ReadWriteOnce @@ -43,77 +37,77 @@ spec: storageClassName: simplyblock-csi-sc ``` -The value is a storage node UUID, found with `{{ cliname }} storage-node list --cluster-id=`. -This is the PVC-level equivalent of `{{ cliname }} volume add --host-id=` (see -[Node Affinity](../../maintenance-operations/node-affinity.md)) — it takes effect regardless of whether the -cluster has node affinity enabled, and always wins over Tier 1 and Tier 2. - -!!! warning "Deprecated annotation prefix" - The `simplybk/` annotation prefix (e.g. `simplybk/host-id`) is deprecated. Existing PVCs using the old - prefix continue to work for backward compatibility, but new deployments should use the - `simplyblock.io/` prefix. - -## Tier 1: Automatic Co-location with the Consuming Pod - -!!! warning "Planned" - Tier 1 is part of a proposed design and is not yet available. This section documents the intended - behavior; check the release notes before relying on it. - -On a cluster with hyper-converged or hybrid nodes, a volume performs best when its primary storage node is -the same worker node the consuming Pod actually lands on — no network hop on the data path. Tier 1 makes -this automatic: it inspects the Pod's *resolved* scheduling outcome (via standard Kubernetes CSI topology, -the same mechanism `WaitForFirstConsumer` StorageClasses already use) and, if that worker also runs a -storage node, stamps the PVC's `host-id` with it before the volume is created. - -Two things must both be true for a given PVC: - -- The storage cluster has [node affinity enabled](../../maintenance-operations/node-affinity.md) - (`--enable-node-affinity` / `StorageCluster.spec.enableNodeAffinity: true`). This is a cluster-wide - precondition — it decides whether co-location is possible at all. -- The PVC carries the opt-in annotation: - - ```yaml title="Requesting co-location with the consuming Pod" - kind: PersistentVolumeClaim - apiVersion: v1 - metadata: - name: my-pvc - annotations: - simplyblock.io/pod-affinity: "true" - spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 10Gi - storageClassName: simplyblock-csi-sc - ``` - -Without this annotation, a PVC on a node-affinity-enabled cluster is *not* automatically co-located — it -falls through to Tier 2/Tier 3 like any other PVC. This lets some workloads request locality while others -on the same cluster participate in ordinary load-aware placement. - -!!! important - This requires a `WaitForFirstConsumer` StorageClass. That's the binding mode that delays provisioning - until a consumer Pod is scheduled — there's no "Pod's resolved node" to co-locate with otherwise. It - also works with any standard scheduling mechanism that ultimately pins the Pod to a node — - `nodeSelector`, node affinity, or pod affinity — but **not** with `spec.nodeName` set directly on the - Pod, since that bypasses the Kubernetes scheduler (and therefore `WaitForFirstConsumer` provisioning) - entirely. This is a general Kubernetes limitation - ([kubernetes/kubernetes#89953](https://github.com/kubernetes/kubernetes/issues/89953)), not specific to - Simplyblock. - -A worker can host more than one storage-node instance (multi-socket deployments). When more than one is -co-located on the Pod's worker, Tier 1 picks among them at random, so volumes spread across every instance -on that worker rather than piling onto one. - -## Opting Out of Automatic Placement - -!!! warning "Planned" - The opt-out annotation below applies once Tier 1 and Tier 2 both exist. Today, only Tier 0 (the - `host-id` pin) needs an explicit annotation to take effect. - -A single PVC can opt out of both automatic tiers (locality and load-aware), regardless of how the cluster -is configured: +The value must be a known storage node UUID, found with `{{ cliname }} storage-node list --cluster-id=`; +a validating webhook rejects the PVC otherwise. On a brand-new PVC, this sets the volume's primary node +directly, before the volume is even created. On an already-bound PVC, setting or changing this same +annotation instead triggers a live migration to the new node — see +[Migrating by Pinning a PVC](../../maintenance-operations/volume-migration.md#migrating-by-pinning-a-pvc). + +!!! warning "Deprecated annotation names" + `simplyblock.io/host-id` and the legacy `simplybk/host-id` prefix are still accepted on a new PVC, but + are rewritten to `simplyblock.io/selected-storage-node` as soon as the PVC is admitted. New deployments + should set `selected-storage-node` directly. + +## Load-Aware Placement for New Volumes + +When [auto-rebalancing's latency benchmark](../../maintenance-operations/volume-migration.md#auto-rebalancing) +is enabled, every new PVC that isn't already pinned is automatically evaluated against the same +per-node latency-deviation signal the rebalancer uses, and stamped with the least-loaded eligible node: + +```yaml title="Enabling load-aware placement for new volumes" +spec: + volumeAutoPlacement: + latencyBenchmarkEnabled: true + prometheusURL: "http://prometheus.simplyblock.svc:9090" +``` + +A node is eligible when it is online, passes its health check, and is below its configured logical volume +limit. Once a node is selected, the operator stamps the PVC with `simplyblock.io/placement-hint` — visible +with `kubectl describe pvc ` — before the volume is created. + +!!! note + A placement hint is a one-shot suggestion, not a pin. The resulting volume remains eligible for + ordinary auto-rebalancing afterwards, unlike a `selected-storage-node` pin. + +## Co-locating a Volume with Its Consuming Pod + +On a hyper-converged or hybrid cluster, a volume performs best when its primary node is the same worker +node the consuming Pod is scheduled onto — no network hop on the data path. A PVC opts into this with: + +```yaml title="Requesting co-location with the consuming Pod" +kind: PersistentVolumeClaim +apiVersion: v1 +metadata: + name: my-pvc + annotations: + simplyblock.io/pod-affinity: "true" +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 10Gi + storageClassName: simplyblock-csi-sc +``` + +This relies on standard Kubernetes CSI topology, so the referenced [StorageClass](storage-class.md) must +use `volumeBindingMode: WaitForFirstConsumer` — that's the binding mode that delays provisioning until a +consumer Pod is scheduled, which is what gives the CSI driver a Pod-resolved node to co-locate against in +the first place. It works no matter which scheduling mechanism placed the Pod there — `nodeSelector`, node +affinity, or pod affinity — but not with `spec.nodeName` set directly on a Pod, since that bypasses the +Kubernetes scheduler (and therefore `WaitForFirstConsumer` provisioning) entirely. This is a general +Kubernetes limitation ([kubernetes/kubernetes#89953](https://github.com/kubernetes/kubernetes/issues/89953)), +not specific to Simplyblock. + +Co-location only takes effect when the volume isn't already pinned or covered by a placement hint (see +above). On a worker node that hosts more than one storage-node instance (multi-socket deployments), the +volume is placed on one of them at random, spreading volumes across every co-located instance instead of +piling onto one. + +## Opting Out + +A single PVC can skip both load-aware placement and Pod co-location, regardless of how the cluster is +configured: ```yaml title="Disabling automatic placement for one PVC" kind: PersistentVolumeClaim @@ -131,11 +125,11 @@ spec: storageClassName: simplyblock-csi-sc ``` -This is different from a `host-id` pin: a pin says "put it exactly here"; this annotation says "don't guess -for me" — the volume falls straight through to the cluster's normal default placement (Tier 3). +This is different from a `selected-storage-node` pin: a pin says "put it exactly here"; this annotation +says "don't choose for me either" — the volume falls straight through to the cluster's built-in default +placement. ## Clones and Snapshot Restores -None of the above applies to a PVC created from a `VolumeSnapshot` or another PVC (`dataSource`) — a -clone or restore always lands on its source volume's own node, unconditionally. Annotations from this page -have no effect on those PVCs. +None of the above applies to a PVC created from a `VolumeSnapshot` or another PVC (`dataSource`) — a clone +or restore always lands on its source volume's own node, unconditionally. From bcdb1cd517eed132889792f227f2c47461a5b992 Mon Sep 17 00:00:00 2001 From: Manohar Reddy Date: Mon, 3 Aug 2026 10:41:04 +0200 Subject: [PATCH 3/4] Rework volume placement page into reference style Matches the terse, table-driven format used elsewhere (e.g. quality-of-service.md, backup-recovery.md's annotation reference, reference/kubernetes/index.md): a resolution-order table, a single annotations reference table, then one compact example per annotation instead of narrative walkthroughs. Co-Authored-By: Claude Sonnet 5 --- .../usage/simplyblock-csi/volume-placement.md | 152 +++++++----------- 1 file changed, 58 insertions(+), 94 deletions(-) diff --git a/docs/usage/simplyblock-csi/volume-placement.md b/docs/usage/simplyblock-csi/volume-placement.md index d725dd54..f0030ecb 100644 --- a/docs/usage/simplyblock-csi/volume-placement.md +++ b/docs/usage/simplyblock-csi/volume-placement.md @@ -1,80 +1,47 @@ --- title: "Automatic Volume Placement" -description: "How Simplyblock chooses a new volume's primary storage node: an explicit pin, automatic load-aware placement, or co-location with the consuming Pod — and how to control each with PVC annotations." +description: "Reference for the PVC annotations and StorageCluster fields that control which storage node becomes a new volume's primary node." weight: 40050 --- -When a new PersistentVolumeClaim is provisioned, Simplyblock decides which storage node becomes the -volume's primary node. Up to three signals can apply, evaluated in this order — the first one available -wins: +Simplyblock resolves a new volume's primary storage node from up to three PVC annotations, evaluated in a +fixed order. If none apply, the storage cluster's built-in default placement is used. -| Order | Signal | Set via | Set by | -|-------|------------------------|-------------------------------------------------|-------------------------| -| 1 | Explicit pin | `simplyblock.io/selected-storage-node` | You | -| 2 | Load-aware placement | `simplyblock.io/placement-hint` | The operator, automatically | -| 3 | Pod co-location | `simplyblock.io/pod-affinity: "true"` (opt-in) | You | +## Resolution Order -A plain PVC with none of the above falls straight through to the storage cluster's own built-in default -placement, exactly as it does today. +| Order | Annotation | Set by | +|-------|---------------------------------------------------|------------------------| +| 1 | `simplyblock.io/selected-storage-node` | User | +| 2 | `simplyblock.io/placement-hint` | Operator (automatic) | +| 3 | `simplyblock.io/pod-affinity: "true"` (opt-in only) | User | +| — | *(none of the above)* | Storage cluster default placement | -## Pinning a Volume to a Specific Node +## Annotations -Use `simplyblock.io/selected-storage-node` to request a specific storage node UUID directly: +| Annotation | Type | Default | Description | +|--------------------------------------------|-----------------------------|---------|----------------------------------------------------------------------------------------------------------------------------| +| `simplyblock.io/selected-storage-node` | string (storage node UUID) | — | Pins the volume to a specific storage node. On a new PVC, sets the primary node directly. On a bound PVC, triggers a live migration to the new node. | +| `simplyblock.io/host-id` | string (storage node UUID) | — | Deprecated alias for `selected-storage-node`. Normalized into it automatically on admission. | +| `simplybk/host-id` | string (storage node UUID) | — | Deprecated legacy prefix for `host-id`. Normalized the same way. | +| `simplyblock.io/placement-hint` | string (storage node UUID) | — | Written automatically by the operator when load-aware placement selects a node for a new volume. Not user-set, and does not pin the volume. | +| `simplyblock.io/pod-affinity` | boolean | `false` | Opts a PVC into co-location with its consuming Pod's resolved node. Requires a `WaitForFirstConsumer` StorageClass. | +| `simplyblock.io/disable-smart-placement` | boolean | `false` | Disables both load-aware placement and Pod co-location for this PVC. Does not affect an explicit pin. | -```yaml title="PVC pinned to a specific storage node" -kind: PersistentVolumeClaim -apiVersion: v1 -metadata: - name: my-pvc - annotations: - simplyblock.io/selected-storage-node: "4e53efdd-86c9-424f-940c-e437eb6a2e95" -spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 10Gi - storageClassName: simplyblock-csi-sc -``` - -The value must be a known storage node UUID, found with `{{ cliname }} storage-node list --cluster-id=`; -a validating webhook rejects the PVC otherwise. On a brand-new PVC, this sets the volume's primary node -directly, before the volume is even created. On an already-bound PVC, setting or changing this same -annotation instead triggers a live migration to the new node — see -[Migrating by Pinning a PVC](../../maintenance-operations/volume-migration.md#migrating-by-pinning-a-pvc). - -!!! warning "Deprecated annotation names" - `simplyblock.io/host-id` and the legacy `simplybk/host-id` prefix are still accepted on a new PVC, but - are rewritten to `simplyblock.io/selected-storage-node` as soon as the PVC is admitted. New deployments - should set `selected-storage-node` directly. - -## Load-Aware Placement for New Volumes - -When [auto-rebalancing's latency benchmark](../../maintenance-operations/volume-migration.md#auto-rebalancing) -is enabled, every new PVC that isn't already pinned is automatically evaluated against the same -per-node latency-deviation signal the rebalancer uses, and stamped with the least-loaded eligible node: +### `simplyblock.io/selected-storage-node` -```yaml title="Enabling load-aware placement for new volumes" -spec: - volumeAutoPlacement: - latencyBenchmarkEnabled: true - prometheusURL: "http://prometheus.simplyblock.svc:9090" +```bash title="Pin a new PVC to a specific storage node" +kubectl annotate pvc my-pvc -n simplyblock \ + simplyblock.io/selected-storage-node=4e53efdd-86c9-424f-940c-e437eb6a2e95 ``` -A node is eligible when it is online, passes its health check, and is below its configured logical volume -limit. Once a node is selected, the operator stamps the PVC with `simplyblock.io/placement-hint` — visible -with `kubectl describe pvc ` — before the volume is created. - -!!! note - A placement hint is a one-shot suggestion, not a pin. The resulting volume remains eligible for - ordinary auto-rebalancing afterwards, unlike a `selected-storage-node` pin. - -## Co-locating a Volume with Its Consuming Pod +The value must be a known storage node UUID, found with `{{ cliname }} storage-node list --cluster-id=`; +a validating webhook rejects the PVC otherwise. Setting this annotation on an already-bound PVC instead +triggers a [migration](../../maintenance-operations/volume-migration.md#migrating-by-pinning-a-pvc) to the +new node. -On a hyper-converged or hybrid cluster, a volume performs best when its primary node is the same worker -node the consuming Pod is scheduled onto — no network hop on the data path. A PVC opts into this with: +### `simplyblock.io/pod-affinity` -```yaml title="Requesting co-location with the consuming Pod" +```yaml title="Co-locate a new volume with its consuming Pod" kind: PersistentVolumeClaim apiVersion: v1 metadata: @@ -90,46 +57,43 @@ spec: storageClassName: simplyblock-csi-sc ``` -This relies on standard Kubernetes CSI topology, so the referenced [StorageClass](storage-class.md) must -use `volumeBindingMode: WaitForFirstConsumer` — that's the binding mode that delays provisioning until a -consumer Pod is scheduled, which is what gives the CSI driver a Pod-resolved node to co-locate against in -the first place. It works no matter which scheduling mechanism placed the Pod there — `nodeSelector`, node -affinity, or pod affinity — but not with `spec.nodeName` set directly on a Pod, since that bypasses the -Kubernetes scheduler (and therefore `WaitForFirstConsumer` provisioning) entirely. This is a general -Kubernetes limitation ([kubernetes/kubernetes#89953](https://github.com/kubernetes/kubernetes/issues/89953)), -not specific to Simplyblock. +| Requirement | Detail | +|------------------------------------------------|---------------------------------------------------------------------------------------------| +| StorageClass binding mode | `WaitForFirstConsumer` (see [Defining a StorageClass](storage-class.md)) | +| Supported scheduling mechanisms | `nodeSelector`, node affinity, pod affinity | +| Not supported | `spec.nodeName` set directly on the Pod ([kubernetes/kubernetes#89953](https://github.com/kubernetes/kubernetes/issues/89953)) | +| Multiple co-located storage nodes on one worker | One is selected at random | +| Precedence | Only applies when the volume isn't already pinned or covered by a placement hint | -Co-location only takes effect when the volume isn't already pinned or covered by a placement hint (see -above). On a worker node that hosts more than one storage-node instance (multi-socket deployments), the -volume is placed on one of them at random, spreading volumes across every co-located instance instead of -piling onto one. +### `simplyblock.io/disable-smart-placement` -## Opting Out +```bash title="Exclude a single PVC from automatic placement" +kubectl annotate pvc my-pvc -n simplyblock \ + simplyblock.io/disable-smart-placement=true +``` -A single PVC can skip both load-aware placement and Pod co-location, regardless of how the cluster is -configured: +Falls through to the storage cluster's default placement, regardless of cluster-wide configuration. -```yaml title="Disabling automatic placement for one PVC" -kind: PersistentVolumeClaim -apiVersion: v1 -metadata: - name: my-pvc - annotations: - simplyblock.io/disable-smart-placement: "true" +## Load-Aware Placement + +Load-aware placement for new volumes is controlled by the same `StorageCluster` field that also feeds +[auto-rebalancing's latency benchmark](../../maintenance-operations/volume-migration.md#auto-rebalancing): + +| Field | Type | Default | Description | +|------------------------------------------------|--------|---------|------------------------------------------------------------------------------------------------------------| +| `volumeAutoPlacement.latencyBenchmarkEnabled` | bool | `false` | Enables load-aware placement for new volumes, independent of `volumeAutoPlacement.enabled` (continuous rebalancer only). | + +```yaml title="Enabling load-aware placement for new volumes" spec: - accessModes: - - ReadWriteOnce - resources: - requests: - storage: 10Gi - storageClassName: simplyblock-csi-sc + volumeAutoPlacement: + latencyBenchmarkEnabled: true + prometheusURL: "http://prometheus.simplyblock.svc:9090" ``` -This is different from a `selected-storage-node` pin: a pin says "put it exactly here"; this annotation -says "don't choose for me either" — the volume falls straight through to the cluster's built-in default -placement. +A node is eligible when it is online, passes its health check, and is below its configured logical volume +limit. ## Clones and Snapshot Restores None of the above applies to a PVC created from a `VolumeSnapshot` or another PVC (`dataSource`) — a clone -or restore always lands on its source volume's own node, unconditionally. +or restore always uses its source volume's node. From 39522457530ec7cf7de85743c34ab2183e35ee10 Mon Sep 17 00:00:00 2001 From: Manohar Reddy Date: Mon, 3 Aug 2026 15:36:34 +0200 Subject: [PATCH 4/4] docs: scope disable-smart-placement to load-aware placement only MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tier 1 (pod-affinity co-location) is already per-PVC opt-in, the inverse of load-aware placement's cluster-wide opt-out default, so disable-smart-placement never actually affected it in a way users needed to override — not setting pod-affinity already excludes a PVC from co-location. Matches the corresponding fix in simplyblock-operator (csi-driver's coLocateEligible/disableSmartPlacement plumbing reverted to a plain pod-affinity check). Co-Authored-By: Claude Sonnet 5 --- docs/usage/simplyblock-csi/volume-placement.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/usage/simplyblock-csi/volume-placement.md b/docs/usage/simplyblock-csi/volume-placement.md index f0030ecb..b01ee968 100644 --- a/docs/usage/simplyblock-csi/volume-placement.md +++ b/docs/usage/simplyblock-csi/volume-placement.md @@ -25,7 +25,7 @@ fixed order. If none apply, the storage cluster's built-in default placement is | `simplybk/host-id` | string (storage node UUID) | — | Deprecated legacy prefix for `host-id`. Normalized the same way. | | `simplyblock.io/placement-hint` | string (storage node UUID) | — | Written automatically by the operator when load-aware placement selects a node for a new volume. Not user-set, and does not pin the volume. | | `simplyblock.io/pod-affinity` | boolean | `false` | Opts a PVC into co-location with its consuming Pod's resolved node. Requires a `WaitForFirstConsumer` StorageClass. | -| `simplyblock.io/disable-smart-placement` | boolean | `false` | Disables both load-aware placement and Pod co-location for this PVC. Does not affect an explicit pin. | +| `simplyblock.io/disable-smart-placement` | boolean | `false` | Disables load-aware placement for this PVC. Does not affect an explicit pin or `pod-affinity` co-location — those are already opt-in per PVC and need no separate opt-out. | ### `simplyblock.io/selected-storage-node` @@ -72,7 +72,9 @@ kubectl annotate pvc my-pvc -n simplyblock \ simplyblock.io/disable-smart-placement=true ``` -Falls through to the storage cluster's default placement, regardless of cluster-wide configuration. +Suppresses load-aware placement for this PVC, regardless of cluster-wide configuration. Placement then falls +through to whatever's next in the [resolution order](#resolution-order) — an explicit pin, `pod-affinity` +co-location if the PVC also requests it, or the storage cluster's default placement. ## Load-Aware Placement