[GSD-13226] zeContextMakeMemoryResident on a peer device allocates a same-sized host (GTT) buffer — 55 GiB of host RAM for a 2-GPU inference server, and it turns out not to be needed

[TSUMUGI-XE]

Pre-submission Checklist

• I am using the latest GPU driver version (releases)
• I have searched for similar issues and found none

GPU Hardware

2× Intel Arc Pro B70 (Battlemage, BMG G31, 8086:e223), behind a PLX PEX880xx switch

DRI Devices Information

$ ls -l /dev/dri/ /dev/dri/by-path/
crw-rw----+ 1 root video  226,   0  card0
crw-rw----+ 1 root video  226,   1  card1
crw-rw----+ 1 root video  226,   2  card2
crw-rw----+ 1 root render 226, 128  renderD128
crw-rw----+ 1 root render 226, 129  renderD129
crw-rw----+ 1 root render 226, 130  renderD130

/dev/dri/by-path:
pci-0000:00:02.0-card    -> ../card0        (iGPU, i915)
pci-0000:00:02.0-render  -> ../renderD128
pci-0000:0e:00.0-card    -> ../card1        (B70 #1, xe)
pci-0000:0e:00.0-render  -> ../renderD129
pci-0000:12:00.0-card    -> ../card2        (B70 #2, xe)
pci-0000:12:00.0-render  -> ../renderD130

GPU Detailed Information (lspci output)

$ lspci -nn | grep -iE 'VGA|Display|3D'
00:02.0 VGA compatible controller [0300]: Intel Corporation RocketLake-S GT1 [UHD Graphics 750] [8086:4c8a] (rev 04)
0e:00.0 VGA compatible controller [0300]: Intel Corporation Battlemage G31 [Intel Graphics] [8086:e223]
12:00.0 VGA compatible controller [0300]: Intel Corporation Battlemage G31 [Intel Graphics] [8086:e223]
1a:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2080 Rev. A] [10de:1e87] (rev a1)

Both B70s use the xe kernel driver. The iGPU (i915) and the NVIDIA card (vfio-pci, passed through
to a VM) are not involved in this report.

Driver Version

26.27.39122.11

Installed GPU Driver Packages

intel-opencl-icd          26.27.39122.11-0
libze-intel-gpu1          26.27.39122.11-0
libze-dev                 1.28.2-2            (loader: libze_loader.so.1.28.2)
intel-igc-core-2          2.38.2
intel-igc-opencl-2        2.38.2

Driver Installation Details

Installed from the Intel apt repository via apt/dpkg; not built from source.

Also reproduced on 26.22.38646 and 26.18.38308. For each version I confirmed the intended
libze_intel_gpu.so was actually loaded by the running process (reading /proc/<pid>/maps), rather
than trusting the installed package version.

Linux Distribution

Other (please specify below)

Other Linux Distribution

Ubuntu 26.04 LTS

Kernel Version & Boot Parameters

$ uname -r
7.1.1-p2pwl

Disclosure: locally built kernel (7.1.1-p2pwl), not a stock distribution kernel. It carries one
local patch that whitelists P2P across a consumer root complex. This should be orthogonal to the
report: both B70s sit under a common PCIe switch (their minimal common ancestor is the PEX880xx
fabric), so their peer traffic is switch-local and is permitted by stock kernels without the patch —
the patch only affects P2P that would traverse the CPU root complex. The host-side allocation reported
here is made by the GPU driver on the make-resident call, independent of the P2P path. A stock Ubuntu
HWE kernel (7.0.0-22-generic) is installed; I can confirm the observation there on request.

Relevant boot parameters:

intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction
i915.enable_guc=0 initcall_blacklist=sysfb_init
vfio-pci.ids=10de:1e87,...

pcie_acs_override and vfio-pci.ids are present to split IOMMU groups for the VFIO passthrough of the
NVIDIA GPU; they do not affect the B70s or their memory behaviour.

Actual Behavior

Peer residency doubles host RAM. Making a VRAM allocation resident on a second device creates a
host-side allocation of the same size (accounted as GTT, invisible to VmRSS / ps / free). At
application scale, a 2-GPU tensor-parallel server with all weights in VRAM consumed 55 GiB of host
RAM — about half of a 123 GiB machine — and an LD_PRELOAD shim that skips peer residency cut that
to 1.55 GiB, with byte-identical output and unchanged throughput. Details and controls below.

Raw Level Zero, no SYCL. 1 GiB allocated on device 0, reading per-device /proc/<pid>/fdinfo:

after zeMemAllocDevice(1 GiB, dev0)     0000:0e:00.0  VRAM 0.00  GTT 0.00
                                        0000:12:00.0  VRAM 0.00  GTT 0.00
after MakeMemoryResident(ptr, dev0)     0000:0e:00.0  VRAM 1.00  GTT 0.00
                                        0000:12:00.0  VRAM 0.00  GTT 0.00
after MakeMemoryResident(ptr, dev1)     0000:0e:00.0  VRAM 1.00  GTT 0.00
                                        0000:12:00.0  VRAM 0.00  GTT 1.00   <-- appears here

Control arms — same program, varying only the residency target:

ctx devices | MakeMemoryResident on | device1 GTT | nr_gpu_active
------------+-----------------------+-------------+---------------
     1      | dev0                  |   0.00 GiB  |   +0.00 GiB
     2      | dev0 only             |   0.00 GiB  |   +0.00 GiB
     2      | dev0 + dev1           |   1.00 GiB  |   +1.00 GiB

The host cost tracks the total device allocation 1:1, and scales with the VRAM pool rather than with
model size (55.1 GiB at --gpu-memory-utilization 0.90, 50.7 GiB at 0.80).

The two counters that do see it are /proc/<pid>/fdinfo/<drm fd> → drm-total-gtt and
/sys/devices/system/node/node0/vmstat → nr_gpu_active. nr_foll_pin_* and Mlocked stay flat, so
this is driver-owned system memory rather than pinned user pages.

Expected Behavior

Per @pbalcer in intel/llvm#22873: "It's not expected that an equivalent host physical allocation is
created for every GPU device allocation." Making a VRAM allocation resident on a peer device should not
require a host-side allocation of the same size.

Evidence this is safe to change: I wrote an LD_PRELOAD shim that records the owning device of each
zeMemAllocDevice result and skips zeContextMakeMemoryResident when the target device is not the
owner. Running the same 2-GPU tensor-parallel server twice, identical except for enabling that skip:

	residency as-is	peer residency skipped
host RAM (nr_gpu_active) peak	55.12 GiB	1.55 GiB
available KV cache	6.14 GiB	6.14 GiB
max concurrency @32k	15.52x	15.50x
collectives (world_size = 2)	OK	OK
generated text (2 prompts, greedy)	—	byte-identical

Scope note: this is one workload, and it never calls the explicit peer-access API, so I am not claiming
peer residency is never necessary. What it shows is that on this hardware, for a workload that actively
uses P2P, the host-side allocation was not required for any functionality.

Reproduction Rate

Always reproduces - 100%

Steps to Reproduce

1. Create one Level Zero context spanning both Battlemage GPUs.
2. zeMemAllocDevice(1 GiB) on device 0.
3. zeContextMakeMemoryResident(ctx, dev0, ptr, size) → device 0 VRAM rises by 1 GiB, no host cost.
4. zeContextMakeMemoryResident(ctx, dev1, ptr, size) → device 1 GTT rises by 1 GiB and
   nr_gpu_active rises by 1 GiB.
5. Read /proc/<pid>/fdinfo/<drm fd> per device (drm-total-vram, drm-total-gtt) and
   /sys/devices/system/node/node0/vmstat (nr_gpu_active) between each step.

Note: zeMemAllocDevice is lazily committed, so step 3/4 is required before any of this is observable.
An earlier version of my test omitted zeContextMakeMemoryResident and reported a clean "no mirror" for
both arms, which was wrong — I corrected that in intel/llvm#22873.

Is this a regression?

• Yes, this is a regression - functionality that previously worked is now broken

Not known to be. Reproduces identically on 26.18.38308, 26.22.38646 and 26.27.39122.11.

System Logs / dmesg Output

Nothing is logged. The behaviour is silent — no errors, no warnings. It is only visible in the memory
counters listed above.

Backtrace (if crash or hang occurred)

No crash or hang occurs.

Source Code / Reproducer

I can attach three reproducers on request: raw Level Zero (produces the table above), a SYCL/PyTorch
version, and the residency-skipping LD_PRELOAD shim used for the comparison. Happy to run variants
here — different allocation sizes, more than two devices, explicit per-device contexts, or an
instrumented driver build.

oneAPI Version (if applicable)

2026.0.0 (also 2026.1 installed on the same machine)

Additional Notes

• [SYCL][Level Zero] sycl::malloc_device allocates a 1:1 host-side (GTT) mirror when the context contains more than one device llvm#22873 is the origin thread; @pbalcer asked me to file here. The explanation there — that
  the runtime makes memory resident on all devices to enable efficient P2P — is what told us where to
  look, and turned a problem we had been chasing for days into something actionable the same evening.
• [SYCL][UR][L0] Restrict USM residency to peers with enabled P2P access llvm#21889 adds SYCL_UR_L0_RESTRICT_USM_RESIDENCY_TO_P2P, which looks like the right
  direction, but it is not in any released oneAPI DPC++ we can install (strings over the shipped
  libur_adapter_level_zero*.so finds no occurrence), so released toolchains cannot reach it.
• The same behaviour has been reported from other applications: SYCL: multi-GPU allocations create host-side GTT mirror not visible in VmRSS / free ggml-org/llama.cpp#22116 and
  [XPU] Use Level Zero zeMemAllocDevice to avoid host memory shadowing pytorch/pytorch#180145.
• P2P itself is unaffected by skipping peer residency: device-to-device copies still measure
  28.59 / 28.58 GB/s, i.e. unchanged.

[kgibala]

Hello @TSUMUGI-XE,

Thank you for your contribution.

Could you please follow the Issue Submission Guide and create an issue based on it? You can edit this one.

Thank you.

[TSUMUGI-XE]

@kgibala Thanks — I have rewritten the issue body following the Issue Submission Guide, using the
[Linux GPU Driver Issue] template fields.

Two notes: the labels (Type: Bug, OS: Linux) are applied automatically at creation time, so editing
the body did not add them — you may need to set those manually. And since I was already here, I
re-verified the report against the latest driver (26.27.39122.11) rather than only the version I
originally filed against.

Happy to add anything else the template asks for that I have missed.