Skip to content

Serialize PCI device hierarchy and top-level attribute files per NUMA node#13312

Open
atoniolo76 wants to merge 2 commits into
google:masterfrom
modal-labs:alessio/serialize-pci-numa-topology
Open

Serialize PCI device hierarchy and top-level attribute files per NUMA node#13312
atoniolo76 wants to merge 2 commits into
google:masterfrom
modal-labs:alessio/serialize-pci-numa-topology

Conversation

@atoniolo76
Copy link
Copy Markdown

@atoniolo76 atoniolo76 commented May 28, 2026

Stacks off #13297 and based on #13114. Adds serialization of the PCI device hierarchy in sysfs collected from /sys/bus/pci/devices/, filtering for GPU, InfiniBand NICs, PCI bridge, and NVSwitch device classes only. Exposes NUMA topology files cpumap, cpulist, and distance from /sys/devices/system/node/node{N}/ for all nodes and includes the top-level aggregate files online, possible, has_cpu, has_memory, and has_normal_memory at /sys/devices/system/node/.

Tested and confirmed full line rate bandwidth:
image

Files added:

  • pci_devices.go adds PCIDeviceAttr struct containing address, realpath, class, vendor, device, subsystem IDs, numa_node, local_cpus, and link speed/width information. Filters for GPU/NIC/Bridge/NVSwitch devices by pciClassRelevant(). Reads PCI device attribute files and walks through the /sys/bus/pci/devices/ directory starting at leaf devices and ending at ancestor bridges. Other plumbing for serializing the data and re-building the chroot sysfs tree.
  • numa.go adds NUMANodeData struct for id, cpumap, cpulist, and distance information. Adds NUMAData structure for the online, possible, has_cpu, has_memory, has_normal_memory aggregate files. Reads from /sys/devices/system/node/node{N}/ per-node to build NUMANodeData. Other plumbing for serializing the data and re-building the node directories with the attribute files plus the aggregate files at the /sys/devices/system/node/ level.

Files modified:

  • pkg/sentry/fsimpl/sys/sys.go adds PCIDevicesData and NUMAData fields to the InternalData struct. Calls newPCIDevicesSysfsEntries() and newNUMASysfsEntries().
  • runsc/cmd/chroot.go adds pciDevicesUpdateChroot() and numaUpdateChroot() functions, which are called before chroot while host sysfs is still accessible and passed as args PCIDevicesData and NUMAData into boot.
  • runsc/cmd/boot.go adds calls to DeserializePCIDevicesData() and DeserializeNUMAData() after pivot root.
  • runsc/boot/loader.go adds pciDevicesData and numaData fields to the loader struct.
  • runsc/boot/vfs.go passes pciDevicesData and numaData into mount options to make available for the sysfs filesystem constructor.

@google-cla
Copy link
Copy Markdown

google-cla Bot commented May 28, 2026

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant