Add RDMA sysfs topology construction for gVisor sandboxes#13712
Open
ayushr2 wants to merge 1 commit into
Open
Conversation
Collaborator
Author
Collaborator
Author
|
AI Usage disclaimer: I did use Claude Code to help me with this patch, but:
|
Containers with /dev/infiniband/uverbs* devices need a sysfs surface for device discovery: libibverbs enumerates /sys/class/infiniband_verbs, and NCCL builds its hardware topology by resolving /sys/class symlinks with realpath() and walking the PCI hierarchy. This adds a --rdmaproxy flag which will gate all RDMA support. For now it only wires up the sysfs topology: runsc snapshots the host sysfs closure derived from the spec's uverbs devices (ibdevs, netdevs, PCI ancestor chains, GPU functions, NUMA nodes) before pivot_root, and the sentry constructs a path-identical virtual tree: one canonical /sys/devices/pci... subtree per device, with /sys/class/* and /sys/bus/pci/devices as symlink farms matching the kernel layout. Per-port dynamic state (RoCE GID tables, link state, counters) is served live through targeted read-only bind mounts of the per-ibdev ports subtrees; identity and PCI attributes are a static snapshot, since the collecting process and the constructed sandbox tree do not observe the same sysfs mount (the RDMA netdevs move into the sandbox network namespace only after the sandbox is created). Each PCI function directory also gets a subsystem symlink into a synthesized /sys/bus/pci, and every node's bus gets a pci_bus entry: NCCL classifies a directory as a PCI device by following its subsystem symlink, and aborts topology discovery without it. The verbs device proxy that makes RDMA usable end-to-end is not part of this change; --rdmaproxy is a work in progress and RDMA is not yet functional. Co-authored-by: Alessio Ricci Toniolo <alessio@modal.com>
atoniolo76
reviewed
Jul 17, 2026
| } | ||
| if len(rdmaDirs.busPCIDevices) > 0 { | ||
| if idata.EnableTPUProxyPaths { | ||
| return nil, nil, fmt.Errorf("cannot enable both TPU proxy paths and RDMA sysfs") |
There was a problem hiding this comment.
Does this mean that TPUProxy must be disabled when RDMAProxy is enabled?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Containers with /dev/infiniband/uverbs* devices need a sysfs surface for
device discovery: libibverbs enumerates /sys/class/infiniband_verbs, and
NCCL builds its hardware topology by resolving /sys/class symlinks with
realpath() and walking the PCI hierarchy.
This adds a --rdmaproxy flag which will gate all RDMA support. For now it
only wires up the sysfs topology: runsc snapshots the host sysfs closure
derived from the spec's uverbs devices (ibdevs, netdevs, PCI ancestor
chains, GPU functions, NUMA nodes) before pivot_root, and the sentry
constructs a path-identical virtual tree: one canonical /sys/devices/pci...
subtree per device, with /sys/class/* and /sys/bus/pci/devices as symlink
farms matching the kernel layout. Per-port dynamic state (RoCE GID tables,
link state, counters) is served live through targeted read-only bind mounts
of the per-ibdev ports subtrees; identity and PCI attributes are a static
snapshot, since the collecting process and the constructed sandbox tree do
not observe the same sysfs mount (the RDMA netdevs move into the sandbox
network namespace only after the sandbox is created).
Each PCI function directory also gets a subsystem symlink into a
synthesized /sys/bus/pci, and every node's bus gets a pci_bus entry: NCCL
classifies a directory as a PCI device by following its subsystem symlink,
and aborts topology discovery without it.
The verbs device proxy that makes RDMA usable end-to-end is not part of
this change; --rdmaproxy is a work in progress and RDMA is not yet
functional.
Co-authored-by: Alessio Ricci Toniolo alessio@modal.com