Add minimal plumbing to serialize sysfs information for ib_write_bw#13297
Open
atoniolo76 wants to merge 1 commit into
Open
Add minimal plumbing to serialize sysfs information for ib_write_bw#13297atoniolo76 wants to merge 1 commit into
atoniolo76 wants to merge 1 commit into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Contributor
|
Very Nice, thank you. |
… ib_write_bw and reconstruct in container. Gate on RDMAProxy config flag.
27977f3 to
d34105a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Based on #13114, this PR provides the minimal devices + sysfs information necessary to get ib_write_bw working between two gVisor containers (relies on later rdmaproxy PRs):
The serialized JSON fields per device contain basic device identity information like
name,abi_version,node_type,node_guid,dev,ibdev,fw_ver, andsys_image_guid. Modalias provides the PCI information that libibverbs uses to match a device to a user space driver plugin (e.g.mlx5). PCI attributes about the NIC device includepci_slot_name,pci_driver,pci_class,pci_vendor,pci_device,pci_subsys_vendor, andpci_subsys_device.numa_nodeandlocal_cpulistdetail which NUMA node and CPUs the NIC is closest to. Finally, theportsarray includes various per-port information.verbs_abi_versiontells libibverbs which version of the kernel uverbs API is supported. This file lives globally insys/class/infiniband_verbs/abi_version.Files added:
pkg/sentry/fsimpl/sys/rdma.gocontain the RDMA-specific data-types, host sysfs collection, and interface for serialization -> deserialization -> reconstruction of virtual sysfs tree.runsc/specutils/rdma.gogates the entire serialization process on the presence of/dev/infiniband/uverbs*in the OCI spec.Files modified:
runsc/cmd/chroot.goexecutes collection of RDMA information from sysfs beforepivot_root. Data is serialized to/var/lib/gvisor/rdma_data.jsoninside the chroot.runsc/cmd/boot.godeserializes the collected data from/var/lib/gvisor/rdma_data.jsonand is passed into the boot argsRDMADevices.runsc/boot/loader.gostores theRDMADevicesargs on theLoaderstruct.runsc/boot/vfs.gopassesrdmaDevicesintosys.InternalData.pkg/sentry/fsimpl/sys/sys.goreadssys.InternalDataand callsrdma.goto build the/sys/class/infiniband_verbs/and/sys/class/infiniband/trees.runsc/container/container.gotells the gofer to create RDMA devices when the OCI spec lists uverbs devices.runsc/cmd/sandboxsetup/gofer_mount.goaddsShouldExposeRDMADeviceto the gofer's device filer for bind-mounting the uverbs device nodes when/dev/infiniband/uverbs*is included in the OCI spec.