Rescan NVMe namespaces when a published namespace device is missing#1159
Rescan NVMe namespaces when a published namespace device is missing#1159grandeit wants to merge 1 commit into
Conversation
When a namespace is mapped to an already-connected NVMe subsystem, the host only creates its device node in response to an asynchronous notification from the controller. If that notification is missed, the device never appears and NodeStageVolume keeps failing with "no device found for the given namespace". Recover the namespace by issuing "nvme ns-rescan" on the subsystem's controllers: from the AttachNVMeVolume retry loop when the device is missing on an already-connected subsystem, and from NVMe self-healing when a published namespace has no device on the host. The self-healing case mirrors iSCSI self-healing, which already rescans the host for devices. A rescan only adds missing namespaces and never renames existing devices, so it is safe on a connected subsystem. Signed-off-by: Manuel Grandeit <m.grandeit@gmail.com>
|
If you wish to contribute to Trident please complete the necessary steps outlined in the CONTRIBUTING.md file, including the signed CCLA. Sign and submit NetApp's Contributor License Agreement. You must sign and submit the Corporate Contributor License Agreement (CCLA) in order to contribute. |
This is already done. Please check CCLA00059 :) |
|
@grandeit We have observed issues with NVMe where a rescan can cause the host side NVMe to hang due to a bug in Linux Kernel. This is a corner case but when the issue is hit the whole node becomes unusable for further workloads. The RHEL Jira defect for this nvme rescan hang issue is tracked at [https://redhat.atlassian.net/browse/RHEL-86484]. Which means we cannot go with this changes as this might pose problem for customers who are not on RHEL9.7 or the corresponding fixed version in other flavours. We are looking for a better solution/design change from our end. |
3 participants
Change description
Rescan NVMe namespaces when a published namespace device is missing
When a namespace is mapped to an already-connected NVMe subsystem, the
host only creates its device node in response to an asynchronous
notification from the controller. If that notification is missed, the
device never appears and NodeStageVolume keeps failing with "no device
found for the given namespace".
Recover the namespace by issuing "nvme ns-rescan" on the subsystem's
controllers: from the AttachNVMeVolume retry loop when the device is
missing on an already-connected subsystem, and from NVMe self-healing
when a published namespace has no device on the host. The self-healing
case mirrors iSCSI self-healing, which already rescans the host for
devices. A rescan only adds missing namespaces and never renames
existing devices, so it is safe on a connected subsystem.
Project tracking
External community contribution. NetApp KB "Unable to mount PVC due to Kubernetes namespace error" documents this exact error but blames a missing namespace. The error fires during node staging, after the controller Publish has already mapped the namespace on ONTAP, so the namespace exists; the host just never enumerated it (nvme ns-rescan recovers it). This PR fixes that cause.
Do any added TODOs have an issue in the backlog?
None added.
Did you add unit tests? Why not?
The behavior that matters, whether the host re-enumerates the namespace after the rescan, is host/kernel behavior a unit test can't reproduce; a test could only assert that nvme ns-rescan is issued, not that the device returns. That recovery is covered by functional testing. Existing utils/nvme tests pass.
Does this code need functional testing?
Yes. This is host-level NVMe/TCP behavior unit tests can't fully cover. Best validated on NVMe/TCP with raw-block volumes by reproducing the hang under concurrent VM clones. Already confirmed manually that nvme ns-rescan recovers the stuck namespace on the affected node.
Is a code review walkthrough needed? why or why not?
A short one would help. The root cause (a missed NVMe discovery notification on an already-connected, namespace-dense subsystem) is non-obvious, and the change adds a new host command and a self-healing remediation.
Should additional test coverage be executed in addition to pre-merge?
NVMe/TCP with raw-block volumes (many namespaces in one subsystem) under concurrent volume creation/clone, plus a check that NodeStage and NodeUnstage are otherwise unaffected.
Does this code need a note in the changelog?
Yes:
Does this code require documentation changes?
No. Internal recovery mechanism, no config, CRD, or API change.
Additional Information
Under concurrent VM clones on NVMe/TCP, PVCs intermittently hang in NodeStageVolume with "no device found for the given namespace" while sibling volumes on the same subsystem mount fine. For raw-block volumes Trident packs many namespaces into one shared subsystem (getSuperSubsystemName, up to 1024), so after the first volume connects it, discovery of each later namespace depends on that one controller's notifications, and a clone burst makes a missed one likely. Trident's only recovery was a 20s retry that re-reads stale sysfs, with no rescan fallback. The self-healing addition mirrors iSCSI self-healing, which already rescans the host for LUNs (scanForAllLUNs).
This is the error from NetApp KB "Unable to mount PVC due to Kubernetes namespace error", which blames a missing namespace and says to recreate the PVC. That cause is wrong here: the error fires after Publish has already mapped the namespace on ONTAP, so it exists, and nvme ns-rescan recovers it (impossible if it didn't exist). Recreating the PVC only works by forcing re-enumeration, at the cost of destroying the PVC's data (e.g. a KubeVirt VM disk). This PR rescans to recover the namespace directly; if one is genuinely absent the rescan is a harmless no-op and the existing error still surfaces.
Builds for linux and darwin, go vet clean, existing utils/nvme tests pass.