Implement safe wrapper for nvmlDeviceGetGpuFabricInfoV.

[mattnappo]

This PR provides a safe wrapper over nvmlReturn_t nvmlDeviceGetGpuFabricInfoV ( nvmlDevice_t device, nvmlGpuFabricInfoV_t* gpuFabricInfo ) by invoking the FFI binding provided by nvml-wrapper-sys in a similar way as the rest of this crate's wrappers.

This returns a Rust struct corresponding to nvmlGpuFabricInfo_v3_t:

unsigned int  cliqueId
// ID of the fabric clique to which this GPU belongs.
unsigned char  clusterUuid[NVML_GPU_FABRIC_UUID_LEN]
// Uuid of the cluster to which this GPU belongs.
unsigned int  healthMask
// GPU Fabric health Status Mask. See NVML_GPU_FABRIC_HEALTH_MASK_*.
unsigned char  healthSummary
// GPU Fabric health summary. See NVML_GPU_FABRIC_HEALTH_SUMMARY_*.
nvmlGpuFabricState_t state
// Current Probe State of GPU registration process. See NVML_GPU_FABRIC_STATE_*.
nvmlReturn_t status
// Probe Error status, if any. Must be checked only if Probe state returns "complete".
unsigned int  version
// Structure version identifier (set to nvmlGpuFabricInfo_v2).

One limitation is that the nvml-wrapper-sys crate does not contain bindings for the following health variants:

• Incompatible GPU Firmware
• Invalid Location
• GPU State Invalid

This can be addressed in follow up work. See

nvml-wrapper/nvml-wrapper-sys/src/bindings.rs

Line 468 in eb47417

pub const NVML_GPU_FABRIC_HEALTH_MASK_DEGRADED_BW_NOT_SUPPORTED: u32 = 0;

The older nvmlReturn_t nvmlDeviceGetGpuFabricInfo ( nvmlDevice_t device, nvmlGpuFabricInfo_t* gpuFabricInfo ) does not provide the healthSummary field, so I do not implement it here (see nvmlGpuFabricInfo_t).

Note that the output of GpuFabricInfo::registration_result is only worth reading when GpuFabricState is Completed. Registering a device with the NVLink fabric is async, and status is only reported when registration/initialization completes.

From the Nvidia docs:

GPU Fabric information
State
Indicates the state of the GPU's handshake with the nvidia-fabricmanager (a.k.a. GPU fabric probe)
Possible values: Completed, In Progress, Not Started, Not supported
Status
Status of the GPU fabric probe response from the nvidia-fabricmanager.
Possible values: NVML_SUCCESS or one of the failure codes.
Clique ID
A clique is a set of GPUs that can communicate to each other over NVLink.
The GPUs belonging to the same clique share the same clique ID.
Clique ID will only be valid for NVLink multi-node systems.
Cluster UUID
UUID of an NVLink multi-node cluster to which this GPU belongs.
Cluster UUID will be zero for NVLink single-node systems.
Health
Summary - Summary of Fabric Health <Healthy, Unhealthy, Limited Capacity>
Bandwidth - is the GPU NVLink bandwidth degraded <Degraded/Full>
Route Recovery in progress - is NVLink route recovery in progress <True/False>
Route Unhealthy - is NVLink route recovery failed or aborted <True/False>
Access Timeout Recovery - is NVLink access timeout recovery in progress <True/False>
Incorrect Configuration - Incorrect Configuration status <Incorrect SystemGuid, Incorrect Chassis Serial Number, No Partition, Insufficient Nvlink Resources, Incompatible GPU Firmware, Invalid Location, GPU State Invalid, None>
Partition Assigned - is the GPU NVLink partition correctly assigned <True/False>

On single-node 8xB300 systems, nvidia-smi reports the following when a device's NVLink is healthy:

    Fabric
        State                             : Completed
        Status                            : Success
        CliqueId                          : 0
        ClusterUUID                       : 00000000-0000-0000-0000-000000000000
        Health
            Summary                       : Healthy
            Bandwidth                     : N/A
            Route Recovery in progress    : N/A
            Route Unhealthy               : N/A
            Access Timeout Recovery       : False
            Incorrect Configuration       : N/A

This can now be represented in Rust like so:

        let raw = nvmlGpuFabricInfoV_t {
            version: 0,
            clusterUuid: [0; 16],
            status: nvmlReturn_enum_NVML_SUCCESS,
            cliqueId: 0,
            state: NVML_GPU_FABRIC_STATE_COMPLETED as u8,
            healthMask: health_mask,
            healthSummary: NVML_GPU_FABRIC_HEALTH_SUMMARY_HEALTHY as u8,
        };

        let info = GpuFabricInfo::from(raw);

During fabric-manager initialization, nvidia-smi reports the following:

Fabric
    State                             : In Progress
    Status                            : N/A
    CliqueId                          : N/A
    ClusterUUID                       : N/A
    Health
        Summary                       : N/A

Machines stuck in this in progress state may have underlying NVLink health issues.

[brayniac]

Thanks for the thorough PR — the version handling, enum mappings, and docs all check out against the NVML 13 header. One real bug to fix before this can merge, though:

The health-mask decoding treats the WIDTH constants as bit counts, but they're bit masks. The header's official accessor is:

#define NVML_GPU_FABRIC_HEALTH_GET(var, type)             \
    (((var) >> NVML_GPU_FABRIC_HEALTH_MASK_SHIFT##type) & \
     (NVML_GPU_FABRIC_HEALTH_MASK_WIDTH##type))

i.e. it masks with & WIDTH directly — WIDTH_DEGRADED_BW = 0x3 and WIDTH_INCORRECT_CONFIGURATION = 0xf are the extraction masks for 2-bit and 4-bit fields. (The shifts confirm this: fields at shifts 0/2/4/6 can only be 2 bits wide without overlapping.)

GpuFabricHealth::from_raw computes (mask >> shift) & ((1u32 << width) - 1), which turns 0x3 into 0x7 and 0xf into 0x7fff, so each field swallows its neighbor's low bit(s).

Concrete failure on a healthy multi-node fabric: healthMask = INCORRECT_CONFIGURATION_NONE << 8 | ACCESS_TIMEOUT_RECOVERY_FALSE << 6 = 0x180, and the current code decodes access_timeout_recovery as (0x180 >> 6) & 0x7 = 6 → GpuFabricHealthFlag::Unknown(6) instead of False. The same contamination hits any 2-bit field whose right-neighbor is TRUE (1). This is invisible on single-node systems like your 8xB300 test machine (everything upstream of bit 8 reads NOT_SUPPORTED = 0), which is presumably why it didn't show up in testing — but multi-node fabrics are exactly where this API matters.

The fix is one line:

let field = |shift: u32, width: u32| (mask >> shift) & width;

Could you also add a unit test with a synthetic mask (e.g. 0x180 → access_timeout_recovery == False, incorrect_config == None) so this stays locked in?

Happy to merge once that's addressed. (Minor non-blocking note: on r535-era drivers that only accept the v2 struct, this call will surface NVML_ERROR_ARGUMENT_VERSION_MISMATCH as NvmlError::UnexpectedVariant(25) since that code isn't mapped in error.rs — a pre-existing gap we can handle in a follow-up.)