Skip to content

Commit ed2b3e4

Browse files
brandboatbrandboat
committed
Add doc for rwx-network
Signed-off-by: Cooper Tseng <cooper.tseng@suse.com>
1 parent 321c68e commit ed2b3e4

4 files changed

Lines changed: 452 additions & 24 deletions

File tree

‎docs/advanced/rwxnetwork.md‎

Lines changed: 365 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,365 @@
1+
---
2+
sidebar_position: 13
3+
sidebar_label: RWX Network
4+
title: "RWX Network"
5+
---
6+
7+
<head>
8+
<link rel="canonical" href="https://docs.harvesterhci.io/v1.6/advanced/rwxnetwork"/>
9+
</head>
10+
11+
Harvester uses Longhorn to provide ReadWriteMany (RWX) volumes for workloads that require shared filesystem access. By default, RWX traffic uses the Kubernetes cluster network. You can configure the [`rwx-network`](./settings.md#rwx-network) setting to either share it with the existing [storage network](./storagenetwork.md), or isolate RWX traffic on a dedicated network for better network bandwidth, performance, and security.
12+
13+
:::note
14+
15+
- Avoid configuring Longhorn settings directly, as this can result in unexpected or unwanted system behavior.
16+
17+
:::
18+
19+
## Prerequisites
20+
21+
Before you begin configuring the RWX network, ensure that the following requirements are met:
22+
23+
- The [cluster network](../networking/clusternetwork.md) and [VLAN network](../networking/harvester-network.md#vlan-network) are configured correctly. Ensure that both networks cover all nodes and are accessible.
24+
25+
- If using **share mode** (`share-storage-network: true`), the [storage network](./storagenetwork.md) must be configured first.
26+
27+
When share mode is enabled, the `longhorn-csi-plugin` and `share-manager` pods also use the storage network and consume IPs from its range. Ensure the storage network IP range accounts for these additional pods. For details, see the [storage network prerequisites](./storagenetwork.md#prerequisites).
28+
29+
- If using **dedicated mode** (`share-storage-network: false` and `network` set), the IP range of the RWX network has the following characteristics:
30+
31+
- Uses the IPv4 CIDR format
32+
33+
- Does not conflict or overlap with Kubernetes cluster networks, storage network, or VM migration network
34+
35+
The following addresses are reserved: `10.42.0.0/16`, `10.43.0.0/16`, `10.52.0.0/16`, and `10.53.0.0/16`.
36+
37+
- Covers the requirements of the cluster
38+
39+
The following pods use the RWX network and each require one IP address:
40+
41+
- `longhorn-csi-plugin` pods: One pod per node (DaemonSet), always present.
42+
- `share-manager` pods: One pod per active RWX volume, created on demand.
43+
44+
The required number of IP addresses is calculated using the following formula: `Required number of IPs = Number of nodes + 32`, where 32 is an arbitrary buffer to accommodate current and future `share-manager` pods.
45+
46+
Example: If a cluster has five nodes, the IP range should be greater than or equal to `/26` (calculation: 5 + 32 = 37).
47+
48+
- Excludes IP addresses that RWX network components must not use, such as addresses reserved for the gateway and other components.
49+
50+
:::caution
51+
52+
During a cluster upgrade, ensure the RWX network IP range has sufficient headroom. Mounted RWX volumes hold their allocated IPs throughout the upgrade, and the upgrade repository deployment uses a Longhorn RWX volume, occupying one additional IP. IP exhaustion during an upgrade can cause it to stall.
53+
54+
:::
55+
56+
- The Whereabouts CNI is installed correctly.
57+
58+
You can check if the `ippools.whereabouts.cni.cncf.io` CRD exists in the cluster using the command `kubectl get crd ippools.whereabouts.cni.cncf.io`.
59+
60+
If an empty string is returned, add the CRDs in [this directory](https://github.com/harvester/harvester/tree/v1.1.0/deploy/charts/harvester/dependency_charts/whereabouts/crds) using the following commands:
61+
62+
```
63+
kubectl apply -f https://raw.githubusercontent.com/harvester/harvester/v1.1.0/deploy/charts/harvester/dependency_charts/whereabouts/crds/whereabouts.cni.cncf.io_ippools.yaml
64+
65+
kubectl apply -f https://raw.githubusercontent.com/harvester/harvester/v1.1.0/deploy/charts/harvester/dependency_charts/whereabouts/crds/whereabouts.cni.cncf.io_overlappingrangeipreservations.yaml
66+
```
67+
68+
- All non-migratable RWX volumes are detached.
69+
70+
You can check the status of RWX volumes using the command:
71+
72+
```bash
73+
kubectl get volumes.longhorn.io -n longhorn-system -o json | jq '.items[] | select(.spec.accessMode == "rwx" and .spec.migratable == false) | {name: .metadata.name, state: .status.state}'
74+
```
75+
76+
## `rwx-network` Setting
77+
78+
The [`rwx-network`](./settings.md#rwx-network) setting allows you to configure the network used to isolate RWX volume traffic when segregation is required.
79+
80+
You can [enable](#enable-the-rwx-network) and [disable](#disable-the-rwx-network) the RWX network using either the UI or the CLI. The setting value is a JSON object with the following structure:
81+
82+
```json
83+
{
84+
"share-storage-network": true|false,
85+
"network": {
86+
"vlan": <vlan-id>,
87+
"clusterNetwork": "<cluster-network-name>",
88+
"range": "<cidr-range>",
89+
"exclude": ["<excluded-ip-or-range>"]
90+
}
91+
}
92+
```
93+
94+
The `network` field is only required when `share-storage-network` is `false`. If `share-storage-network` is `false` and no `network` is specified, RWX traffic uses the Kubernetes default cluster network. This is the default behavior when the `rwx-network` setting is not configured.
95+
96+
The following occur once the `rwx-network` setting is applied:
97+
98+
- Harvester creates or updates the NetworkAttachmentDefinition for RWX traffic.
99+
- Harvester updates the Longhorn `endpoint-network-for-rwx-volume` setting.
100+
- Longhorn restarts all `longhorn-csi-plugin` pods to apply the new network configuration.
101+
102+
### Configuration Steps
103+
104+
<Tabs>
105+
<TabItem value="ui" label="UI" default>
106+
107+
:::tip
108+
109+
Using the Harvester UI to configure the `rwx-network` setting is strongly recommended.
110+
111+
:::
112+
113+
#### Enable the RWX Network
114+
115+
1. Go to **Advanced > Settings > rwx-network**.
116+
117+
1. Select the desired mode:
118+
- **Share Storage Network**: Select this option to reuse the storage network configuration. No additional fields are required.
119+
- **Dedicated Network**: Select this option to configure a separate network for RWX traffic.
120+
121+
1. If using **Dedicated Network** mode, configure the following fields to construct a Multus `NetworkAttachmentDefinition` CRD:
122+
- **VLAN ID**: (Optional) The VLAN ID for RWX network traffic
123+
- **Cluster Network**: The cluster network to use (must be pre-configured)
124+
- **IP Range**: The IPv4 CIDR range for RWX network IPs
125+
- **Exclude**: (Optional) IP addresses or ranges to exclude from allocation
126+
127+
1. Click **Save**.
128+
129+
#### Disable the RWX Network
130+
131+
1. Go to **Advanced > Settings > rwx-network**.
132+
133+
1. Select **Disabled** or reset to the default value.
134+
135+
1. Click **Save**.
136+
137+
Once the RWX network is disabled, Longhorn starts using the pod network for RWX volume operations.
138+
139+
</TabItem>
140+
<TabItem value="cli" label="CLI">
141+
142+
You can use the following command to configure the [`rwx-network`](./settings.md#rwx-network) setting.
143+
144+
```bash
145+
kubectl edit settings.harvesterhci.io rwx-network
146+
```
147+
148+
#### Enable Share Storage Network Mode
149+
150+
```yaml
151+
apiVersion: harvesterhci.io/v1beta1
152+
default: '{"share-storage-network":false}'
153+
kind: Setting
154+
metadata:
155+
name: rwx-network
156+
value: '{"share-storage-network":true}'
157+
```
158+
159+
#### Enable Dedicated Network Mode
160+
161+
```yaml
162+
apiVersion: harvesterhci.io/v1beta1
163+
default: '{"share-storage-network":false}'
164+
kind: Setting
165+
metadata:
166+
name: rwx-network
167+
value: '{"share-storage-network":false,"network":{"vlan":200,"clusterNetwork":"rwx","range":"192.168.1.0/24","exclude":["192.168.1.1/32","192.168.1.254/32"]}}'
168+
```
169+
170+
#### Disable the RWX Network
171+
172+
Remove the `value` field or set it to the default:
173+
174+
```yaml
175+
apiVersion: harvesterhci.io/v1beta1
176+
default: '{"share-storage-network":false}'
177+
kind: Setting
178+
metadata:
179+
name: rwx-network
180+
value: '{"share-storage-network":false}'
181+
```
182+
183+
</TabItem>
184+
</Tabs>
185+
186+
### Post-Configuration Steps
187+
188+
:::info important
189+
190+
After configuring the RWX network, verify that the configuration is correct and applied successfully before resuming RWX workloads.
191+
192+
:::
193+
194+
1. Verify that the `rwx-network` setting's status is `True` and the type is `configured` using the following command:
195+
196+
```bash
197+
kubectl get settings.harvesterhci.io rwx-network -o yaml
198+
```
199+
200+
Example:
201+
202+
```yaml
203+
apiVersion: harvesterhci.io/v1beta1
204+
default: '{"share-storage-network":false}'
205+
kind: Setting
206+
metadata:
207+
annotations:
208+
rwx-network.settings.harvesterhci.io/hash: cc5102f8a0442e4e6d7e13923482f6594333b498
209+
rwx-network.settings.harvesterhci.io/initialized: "true"
210+
rwx-network.settings.harvesterhci.io/net-attach-def: harvester-system/rwx-network-976r9
211+
rwx-network.settings.harvesterhci.io/old-net-attach-def: ""
212+
creationTimestamp: "2026-03-20T01:48:26Z"
213+
generation: 6
214+
name: rwx-network
215+
resourceVersion: "1280504"
216+
uid: 5f934df5-b483-403c-af43-c1f1ec146db4
217+
status:
218+
conditions:
219+
- lastUpdateTime: "2026-03-20T01:58:25Z"
220+
reason: Completed
221+
status: "True"
222+
type: configured
223+
value: '{"share-storage-network":false,"network":{"clusterNetwork":"rwx","range":"192.168.201.0/24","exclude":["192.168.201.1/32"]}}'
224+
```
225+
226+
2. Verify that the Longhorn CSI plugin pods are ready and that their networks are correctly configured.
227+
228+
You can list the Longhorn CSI plugin pods using the following command:
229+
230+
```bash
231+
kubectl get pods -n longhorn-system -l app=longhorn-csi-plugin
232+
```
233+
234+
Inspect each pod to verify the network configuration:
235+
236+
```bash
237+
kubectl -n longhorn-system describe pod <pod-name>
238+
```
239+
240+
Errors similar to the following indicate that the RWX network has exhausted its available IP addresses. You must reconfigure the RWX network with a sufficient IP range.
241+
242+
```bash
243+
Events:
244+
Type Reason Age From Message
245+
---- ------ ---- ---- -------
246+
Warning FailedCreatePodSandBox 2m58s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "xyz123": plugin type="multus" name="multus-cni-network" failed (add): error adding container to network "rwx-network-abc123": error at storage engine: Could not allocate IP in range
247+
```
248+
249+
3. Verify that an interface named `lhnet2` exists in the `k8s.v1.cni.cncf.io/network-status` annotations. The IP address of this interface must be within the designated IP range.
250+
251+
You can retrieve a list of Longhorn CSI plugin pods with network details using the following command:
252+
253+
```bash
254+
kubectl get pods -n longhorn-system -l app=longhorn-csi-plugin -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.metadata.annotations.k8s\.v1\.cni\.cncf\.io/network-status}{"\n"}{end}'
255+
```
256+
257+
Verify that each pod has an interface with the expected IP address from your RWX network range.
258+
259+
Example:
260+
```
261+
longhorn-csi-plugin-x8pxs [{
262+
"name": "k8s-pod-network",
263+
"interface": "eth0",
264+
"ips": [
265+
"10.52.0.59"
266+
],
267+
"mac": "b2:7d:99:bc:d5:dd",
268+
"default": true,
269+
"dns": {}
270+
},{
271+
"name": "harvester-system/rwx-network-qb6zp",
272+
"interface": "lhnet2",
273+
"ips": [
274+
"192.168.201.2"
275+
],
276+
"mac": "ea:9f:a0:48:62:b6",
277+
"dns": {}
278+
}]
279+
```
280+
281+
4. Create a test RWX workload to verify end-to-end functionality.
282+
283+
If you do not already have an RWX StorageClass, create one first:
284+
285+
```yaml
286+
kind: StorageClass
287+
apiVersion: storage.k8s.io/v1
288+
metadata:
289+
name: longhorn-rwx
290+
provisioner: driver.longhorn.io
291+
allowVolumeExpansion: true
292+
reclaimPolicy: Delete
293+
volumeBindingMode: Immediate
294+
parameters:
295+
numberOfReplicas: "3"
296+
staleReplicaTimeout: "2880"
297+
fromBackup: ""
298+
fsType: "ext4"
299+
nfsOptions: "vers=4.2,noresvport,softerr,timeo=600,retrans=5"
300+
```
301+
302+
Then apply the following manifest to create a temporary PVC and pod:
303+
304+
```yaml
305+
apiVersion: v1
306+
kind: PersistentVolumeClaim
307+
metadata:
308+
name: rwx-test-pvc
309+
spec:
310+
accessModes:
311+
- ReadWriteMany
312+
resources:
313+
requests:
314+
storage: 1Gi
315+
storageClassName: longhorn-rwx
316+
---
317+
apiVersion: v1
318+
kind: Pod
319+
metadata:
320+
name: rwx-test-pod
321+
spec:
322+
containers:
323+
- name: test
324+
image: busybox
325+
command: ["sh", "-c", "echo RWX OK > /data/test.txt && cat /data/test.txt && sleep 3600"]
326+
volumeMounts:
327+
- name: data
328+
mountPath: /data
329+
volumes:
330+
- name: data
331+
persistentVolumeClaim:
332+
claimName: rwx-test-pvc
333+
```
334+
335+
Verify the pod reaches `Running` state and the volume is mounted successfully:
336+
337+
```bash
338+
kubectl get pod rwx-test-pod
339+
kubectl logs rwx-test-pod
340+
```
341+
342+
A log output of `RWX OK` confirms that the RWX network is functioning correctly. Clean up the test resources when done:
343+
344+
```bash
345+
kubectl delete pod rwx-test-pod
346+
kubectl delete pvc rwx-test-pvc
347+
```
348+
349+
Once the configuration is verified, RWX volumes can be used by workloads.
350+
351+
## Best Practices
352+
353+
- **Plan IP allocation carefully**: When configuring an IP range for the RWX network in dedicated mode, ensure that the allocated IP addresses can service the future needs of the cluster. The CSI plugin pods are deployed as a DaemonSet (one per node), so you need to account for future node additions.
354+
355+
- **Choose the appropriate mode**:
356+
- Use **share mode** if you want to consolidate storage-related traffic and simplify network management.
357+
- Use **dedicated mode** if you need different network characteristics (bandwidth, QoS, security policies) for RWX volumes compared to storage replication traffic.
358+
359+
- **Avoid mgmt cluster network**: Configure the RWX network on a non-`mgmt` cluster network to ensure complete separation from the Kubernetes control plane traffic. Using `mgmt` is possible but not recommended because of the negative impact (resource and bandwidth contention) on the control plane network performance.
360+
361+
- **Monitor network utilization**: After configuring the RWX network, monitor the network utilization to ensure adequate bandwidth for your RWX workloads.
362+
363+
- **Test before production**: Test the RWX network configuration in a non-production environment before applying it to production workloads.
364+
365+
- **Plan maintenance windows**: Changing the RWX network configuration requires restarting CSI plugin pods, which will temporarily disrupt operations that depend on the CSI plugin, including VM disk provisioning, volume attachment and detachment. Plan for appropriate maintenance windows.

0 commit comments

Comments
 (0)