vmm: sanitize GPUs via VFIO hot reset instead of sysfs SBR

[Leechael]

Summary

• switch GPU sanitize-on-attach (vmm: sanitize GPUs with SBR before QEMU attach #1048) from root-only sysfs writes to the VFIO_DEVICE_PCI_HOT_RESET ioctl so it works with the VMM running as an unprivileged user
• validate the kernel-reported reset scope: every device affected by the reset must belong to the GPU's own IOMMU group, otherwise the launch is aborted
• drop the /sys/bus/pci/drivers_probe re-probe; devices stay bound to vfio-pci across the reset
• add a dstack-vmm sanitize-gpu <slot>... subcommand for manually resetting GPUs through the exact launch-time code path

Motivation

#1048 cannot be enabled in a common operations setup: dstack-vmm running as a dedicated unprivileged user without sudo. It issues the Secondary Bus Reset by writing Bridge Control in the upstream bridge's sysfs config space and re-probes devices through /sys/bus/pci/drivers_probe, and both files are writable by root only.

VFIO_DEVICE_PCI_HOT_RESET makes the kernel perform the same Secondary Bus Reset on the parent bridge, but it is authorized by device ownership instead of privilege: the caller presents an fd for every VFIO group affected by the reset. The /dev/vfio group nodes are the same ones QEMU opens to attach the GPU, so the VMM user already has access to them. This is also the mechanism QEMU itself uses to bus-reset devices without a usable FLR.

A single group fd suffices because every sanitized GPU sits alone behind a dedicated PCIe bridge and alone in its IOMMU group. This matches the typical passthrough host setup: H200-class GPUs behind dedicated downstream ports, one device per IOMMU group, vfio-pci bound at boot, and the VMM user granted /dev/vfio access via the same group membership QEMU requires. A read-only survey of production H200 hosts confirmed this topology and access model.

The privilege constraints and topology preconditions are documented at the top of gpu_reset.rs.

Safety

• the dedicated-bridge topology check from vmm: sanitize GPUs with SBR before QEMU attach #1048 is retained as defense
• VFIO_DEVICE_PCI_GET_HOT_RESET_INFO output is validated before resetting: any affected device outside the GPU's own IOMMU group aborts the launch, and the affected set is logged
• all VFIO fds are closed before QEMU is spawned; a VFIO group can be opened by only one process at a time
• the ioctl either performs the bus reset or fails; it cannot silently degrade to FLR

Tests

• cargo check -p dstack-vmm; cargo clippy clean
• cargo test -p dstack-vmm (117 passed)
• not yet validated on GPU hardware — no test GPU is currently available. Planned validation once one is: a kprobe on pci_bridge_secondary_bus_reset to confirm the kernel executes the SBR on the expected bridge, config-space polling for the link-down window during reset, an A/B comparison against the sysfs method, and a reproduction of the original SPDM-timeout incident. sanitize_on_attach stays config-gated until then, and dstack-vmm sanitize-gpu runs the same path standalone for the experiment.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.