Tesla V100 16G PCIe on Dell Precision 7920T

[Donalaren]

Hello, I am in urgent need of your assistance. I am attempting to use a Tesla V100 16G PCIe graphics card on a Dell Precision 7920T.

System
Motherboard: Dell Precision 7920T
Card: Tesla V100 16G PCIe
System: Ubuntu 22.04
RAM: 64G

• Memory Map IO above 4GB is enabled.
• There is no CSM in BIOS of Dell 7920T.
• BIOS version is updated (2.50.0).

Description
I first attempted this on Ubuntu 22.04 using @xCuri0 's ReBarState. Followed this method, after building and running RebarState, set the value to 32. But this approach proved ineffective. After rebooting and reopening the RebarState, I can see the Current ReBarState 32/Unlimited. nvidia-smi still can't see my Tesla V100 ( lspci | grep -i nvidia correctly identified the GPU model 0000:73:00.0 3D controller: NVIDIA Corporation GV100GL [Tesla V100 PCIe 16GB] (rev a1)).
Through dmesg I can see:

[   45.605811] nvidia 0000:73:00.0: enabling device (0140 -> 0142)
[   45.606250] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
               NVRM: BAR1 is 0M @ 0x0 (PCI:0000:73:00.0)
[   45.606542] nvidia: probe of 0000:73:00.0 failed with error -1

This is consistent with the situation without ReBarState. I have only followed this method to build and run ReBarState on the Ubuntu system without making any modifications to the BIOS. My actions were as follows:

git clone https://github.com/xCuri0/ReBarUEFI.git
cd ReBarUEFI/ReBarState
mkdir build
cd build
cmake ..
make
sudo ./ReBarState

So I suspect ReBarState may not be functioning. How to verify if the RebarState is working in this scenario?
Note that I'm not suspecting the software itself; I believe my execution may be incorrect. Any suggestions would be most gratefully received.

[xCuri0]

@Donalaren Tesla V100 and all NVIDIA datacenter GPUs don't use Resizable BAR they just has a large BAR by default, the issue here is likely with your BIOS configuration.

Dell 7920T should be recent enough to not have the weird issues on older motherboards requiring UEFIPatches.

All ReBarState does is read/write the ReBarState EFI variable which you could also read yourself but it's going to give the same result as what you're seeing.

EDIT: Seems like you aren't running ReBarUEFI modded BIOS even, in that case ReBarState does absolutely nothing.

[Donalaren]

Thank you very much for your reply; it has given me renewed hope.

Do you believe RebarState cannot resolve this compatibility issue with the Tesla V100? Yet I've observed on other websites that they've addressed Tesla GPU problems (including Tesla P100 and Tesla P40). Moreover, they all mention switching the GPU mode to graphics mode via NVFlash. However, NVFlash currently fails to detect my Tesla GPU, I suspect this stems from the absence of Rebar, as per this webpage. Regardless, Rebar is essential, so I intend to give it a try.

Thus, we return to the RebarState software. I acknowledge it is an extremely useful tool—thank you for sharing it—but I seem to have yet to grasp how to utilise it properly. Clearly, my understanding of this statement was flawed. I mistakenly believed that , I only needed to run ReBarState without modifying BIOS on the Ubuntu system.
Right, I understand now. Even if I wish to implement Resizable BAR on a Linux system, I still need to flash the modified BIOS using hardware BIOS flasher such as CH341A. However, I encountered difficulties when carrying out the first step Adding FFS module. Dell's BIOS file is in .exe format, which UEFITool cannot recognise. I've observed that some Dell workstations have successfully implemented Rebar in List of tested motherboards. How might one resolve this issue about the EXE format?

[xCuri0]

Tesla V100 isn't a rebar GPU it's going to make no difference.

[xCuri0]

I'm surprised a workstation like Dell 7920T has issues like this because regular LGA1151/1150 boards can support 32GB BAR with no issues. And even LGA1155 can support it with reduced (<32GB) RAM installed.

I believe this is a BIOS configuration issue and even if it can't be fixed that way it can through use of Linux kernel command line https://docs.kernel.org/admin-guide/kernel-parameters.html CTRL+F "PCI Subsystem" and/or DSDT editing

[Donalaren]

Thank you for your suggestion. Yes, I asked my Dell sales representative, who has more hands-on experience with this machine. They explicitly told me that the Tesla graphics card is not compatible with this workstation. I also couldn't find any successful compatibility cases online for Dell 7920T and Tesla V100.

1. Given this, I believe the BIOS configuration should be fine. And, I carefully reviewed the BIOS settings and found no other relevant configurations besides the “Above 4G” option. Attached is my BIOS setup guide.
   precision-7920-tower.pdf

2. I've tried pci=realloc too, but I still get the same error

[ 2175.144379] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:
               NVRM: BAR1 is 0M @ 0x0 (PCI:0000:73:00.0)
[ 2175.144672] nvidia: probe of 0000:73:00.0 failed with error -1
[ 2175.144767] NVRM: The NVIDIA probe routine failed for 1 device(s).
[ 2175.144771] NVRM: None of the NVIDIA devices were initialized.
[ 2175.146208] nvidia-nvlink: Unregistered Nvlink Core, major device number 235

3. Next, I might start by researching DSDT editing.

[xCuri0]

Have you looked into IntelRCSetup ? An issue similar to this is mentioned in common issues and can be fixed in it https://github.com/xCuri0/ReBarUEFI/wiki/Common-issues-(and-fixes)#system-wont-boot-with-large-vram-16gb-or-multi-gpu

I recommend not to try DSDT patching because PCIe is way more complicated on HEDT/server boards which makes it less likely to work.

[Donalaren]

There is no IntelRCSetup in the BIOS of Dell 7920T. I tested the consumer-grade GeForce RTX 4070 and successfully got it working on the 7920T, which is easy. Additionally, the Tesla V100 was successfully configured on the MSI Z790 motherboard. In comparison, the Dell motherboard has significantly limited functionality, including graphics card configuration and manual fans configuration, despite potentially offering higher reliability.

[xCuri0]

Are you able to share a full BIOS dump ? I could see if there's any hidden settings in it because there's no way a 2017 workstation doesn't support large BAR.

C621 chipset should support 4G decoding with no issues and I think some have even been updated to have resizable BAR.

[Icbears]

@Donalaren This information is pretty important to me because I am also looking for a V100 for my 7820T, and they have the same chipset and pcie slots configurations. I noticed that someone tried to use V100 on a 7920T with Windows 10, while he came with the error code (12), which said the V100 does not have enough resources to use. So, I am wondering if it is possible to allocate more pcie bus to slot 2 or 4 where the V100 goes on could help. You may try to get into bios setup->advanced configurations->pcie bus->change into option 1 (with more bus allocated to pcie slot 2 and 4), then boot up to see if the problem can be solved.