nvproxy: support exporting GPU memory to a dma-buf fd#13736
Open
ayushr2 wants to merge 2 commits into
Open
Conversation
Containers with /dev/infiniband/uverbs* devices need a sysfs surface for device discovery: libibverbs enumerates /sys/class/infiniband_verbs, and NCCL builds its hardware topology by resolving /sys/class symlinks with realpath() and walking the PCI hierarchy. This adds a --rdmaproxy flag which will gate all RDMA support. For now it only wires up the sysfs topology: runsc snapshots the host sysfs closure derived from the spec's uverbs devices (ibdevs, netdevs, PCI ancestor chains, GPU functions, NUMA nodes) before pivot_root, and the sentry constructs a path-identical virtual tree: one canonical /sys/devices/pci... subtree per device, with /sys/class/* and /sys/bus/pci/devices as symlink farms matching the kernel layout. Per-port dynamic state (RoCE GID tables, link state, counters) is served live through targeted read-only bind mounts of the per-ibdev ports subtrees; identity and PCI attributes are a static snapshot, since the collecting process and the constructed sandbox tree do not observe the same sysfs mount (the RDMA netdevs move into the sandbox network namespace only after the sandbox is created). Each PCI function directory also gets a subsystem symlink into a synthesized /sys/bus/pci, and every node's bus gets a pci_bus entry: NCCL classifies a directory as a PCI device by following its subsystem symlink, and aborts topology discovery without it. The verbs device proxy that makes RDMA usable end-to-end is not part of this change; --rdmaproxy is a work in progress and RDMA is not yet functional. Co-authored-by: Alessio Ricci Toniolo <alessio@modal.com>
Add support for NV_ESC_EXPORT_TO_DMABUF_FD, which CUDA issues to export GPU memory as a dma-buf file descriptor. This is the mechanism used for GPUDirect RDMA: the returned fd is handed to ibv_reg_dmabuf_mr so a NIC can DMA directly to and from GPU memory. The driver allocates a host dma-buf and returns its fd in the ioctl's FD field; nvproxy wraps that host fd in a sentry file description installed in the application's fd table. The wrapper implements vfs.HostFDProvider so the RDMA device proxy can recover the host fd for registration. This ioctl exposes GPU memory to external devices, so it is gated behind a new privileged driver capability, CapRDMA, and is only permitted (and its seccomp filter only installed) when that capability is enabled. CapRDMA is enabled automatically when RDMA support is requested. Co-authored-by: Alessio Ricci Toniolo <alessio@modal.com>
Collaborator
Author
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add support for NV_ESC_EXPORT_TO_DMABUF_FD, which CUDA issues to export
GPU memory as a dma-buf file descriptor. This is the mechanism used for
GPUDirect RDMA: the returned fd is handed to ibv_reg_dmabuf_mr so a NIC
can DMA directly to and from GPU memory.
The driver allocates a host dma-buf and returns its fd in the ioctl's FD
field; nvproxy wraps that host fd in a sentry file description installed
in the application's fd table. The wrapper implements vfs.HostFDProvider
so the RDMA device proxy can recover the host fd for registration.
This ioctl exposes GPU memory to external devices, so it is gated behind
a new privileged driver capability, CapRDMA, and is only permitted (and
its seccomp filter only installed) when that capability is enabled.
CapRDMA is enabled automatically when RDMA support is requested.
Co-authored-by: Alessio Ricci Toniolo alessio@modal.com