Skip to content

[GSD-13189] Hang on exit in llama.cpp after memory overcommit on Intel B70 #967

Description

@rriggs

Pre-submission Checklist

  • I am using the latest GPU driver version (releases)
  • I have searched for similar issues and found none

GPU Hardware

Intel Arc Pro B70

DRI Devices Information

$ ls -ls /dev/dri/*
0 crw-rw----+ 1 root video  226,   1 Aug  3 17:48 /dev/dri/card1 
0 crw-rw----+ 1 root video  226,   2 Aug  3 17:48 /dev/dri/card2 
0 crw-rw-rw-. 1 root render 226, 128 Aug  2 21:43 /dev/dri/renderD128 
0 crw-rw-rw-. 1 root render 226, 129 Aug  2 21:43 /dev/dri/renderD129 

/dev/dri/by-path:
total 0
0 lrwxrwxrwx. 1 root root  8 Aug  2 21:43 pci-0000:03:00.0-card -> ../card1 
0 lrwxrwxrwx. 1 root root 13 Aug  2 21:43 pci-0000:03:00.0-render -> ../renderD128 
0 lrwxrwxrwx. 1 root root  8 Aug  2 21:43 pci-0000:19:00.0-card -> ../card2 
0 lrwxrwxrwx. 1 root root 13 Aug  2 21:43 pci-0000:19:00.0-render -> ../renderD129 
$ ls -la /dev/dri/by-path/
total 0
drwxr-xr-x. 2 root root 120 Aug  2 21:43 .
drwxr-xr-x. 3 root root 140 Aug  2 21:43 ..
lrwxrwxrwx. 1 root root   8 Aug  2 21:43 pci-0000:03:00.0-card -> ../card1 
lrwxrwxrwx. 1 root root  13 Aug  2 21:43 pci-0000:03:00.0-render -> ../renderD128 
lrwxrwxrwx. 1 root root   8 Aug  2 21:43 pci-0000:19:00.0-card -> ../card2 
lrwxrwxrwx. 1 root root  13 Aug  2 21:43 pci-0000:19:00.0-render -> ../renderD129 

GPU Detailed Information (lspci output)

$ sudo lspci -vvv -k -s 0000:03:00.0
03:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Arc Pro B70] (prog-if 00 [VGA controller])
        Subsystem: Intel Corporation Device 1701                    
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes                                                                                                                  
        Interrupts: unknown pin routed to IRQ 196, MSI(X) routed to IRQ 196                                                                      19:57 [37/605]
        IOMMU group: 16
        Region 0: Memory at f800000000 (64-bit, prefetchable) [size=16M]
        Region 2: Memory at f000000000 (64-bit, prefetchable) [size=32G]
        Expansion ROM at f5c00000 [disabled] [size=2M]
        Capabilities: [40] Vendor Specific Information: Intel Capabilities v1
                CapA: Peg60Dis- Peg12Dis- Peg11Dis- Peg10Dis- PeLWUDis- DmiWidth=x4
                      EccDis- ForceEccEn- VTdDis- DmiG2Dis- PegG2Dis- DDRMaxSize=Unlimited
                      1NDis- CDDis- DDPCDis- X2APICEn- PDCDis- IGDis- CDID=0 CRID=0
                      DDROCCAP+ OCEn- DDRWrtVrefEn+ DDR3LEn+
                CapB: ImguDis- OCbySSKUCap- OCbySSKUEn- SMTCap- CacheSzCap 0x0
                      SoftBinCap- DDR3MaxFreqWithRef100=Disabled PegG3Dis-
                      PkgTyp- AddGfxEn- AddGfxCap- PegX16Dis- DmiG3Dis- GmmDis-
                      DDR3MaxFreq=2932MHz LPDDR3En-
        Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
                LnkSta: Speed 2.5GT/s, Width x1
                        TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                         AtomicOpsCtl: ReqEn-
                         IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
                         10BitTagReq- OBFF Disabled, EETLPPrefixBlk-
                LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
        Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
                Address: 00000000fee00000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [d0] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [110 v1] Null
        Capabilities: [200 v1] Address Translation Service (ATS)
                ATSCap: Invalidate Queue Depth: 00
                ATSCtl: Enable+, Smallest Translation Unit: 00
        Capabilities: [420 v1] Physical Resizable BAR
                BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
        Capabilities: [220 v1] Virtual Resizable BAR
                BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
        Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration- 10BitTagReq+ IntMsgNum 0
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
                IOVSta: Migration-
                Initial VFs: 4, Total VFs: 4, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 1, stride: 1, Device ID: e223
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 0000000000000000 (64-bit, prefetchable)
                Region 2: Memory at 0000000000000000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [400 v1] Latency Tolerance Reporting
                Max snoop latency: 1048576ns
                Max no snoop latency: 1048576ns
        Kernel driver in use: xe
        Kernel modules: xe

Driver Version

26.18.38308.4

Installed GPU Driver Packages

$ sudo rpm -qa | grep -iE "igc|gmm|opencl|level-zero|level_zero|ocloc|libze"
libsigc++30-3.6.0-6.fc43.x86_64
wine-opencl-11.0-2.fc43.x86_64
wine-opencl-11.0-2.fc43.i686
cuda-opencl-12-9-12.9.19-1.x86_64
intel-gmmlib-22.10.0-2.fc43.x86_64
intel-opencl-clang-15.0.9-2.fc43.x86_64
libsigc++20-2.12.2-1.fc43.x86_64
oneapi-level-zero-1.28.6-1.fc43.x86_64
oneapi-level-zero-devel-1.28.6-1.fc43.x86_64
intel-level-zero-gpu-raytracing-1.2.0-2.fc43.x86_64
oneapi-level-zero-zello_world-1.28.6-1.fc43.x86_64
intel-igc-2.34.4-2.fc43.x86_64
intel-igc-libs-2.34.4-2.fc43.x86_64
intel-level-zero-26.18.38308.4-2.fc43.x86_64
intel-ocloc-26.18.38308.4-2.fc43.x86_64
intel-opencl-26.18.38308.4-2.fc43.x86_64

Driver Installation Details

Distribution: Fedora 43

Regalar dnf updates.

Linux Distribution

Fedora / RHEL

Other Linux Distribution

Fedora 43

Kernel Version & Boot Parameters

$ uname -r
7.1.5-101.fc43.x86_64

$ cat /proc/cmdline
BOOT_IMAGE=(hd1,gpt2)/vmlinuz-7.1.5-101.fc43.x86_64 root=UUID=bdad322a-efcb-4f9d-94fc-10fead02837c ro rootflags=subvol=root rhgb quiet elevator=none amdgpu.modeset=1

$ lsmod | grep -E 'i915|xe'
xe                   4993024  8
intel_vsec             28672  2 pmt_telemetry,xe
drm_gpuvm              57344  1 xe
drm_gpusvm_helper      61440  1 xe
drm_buddy              12288  2 amdgpu,xe
i2c_algo_bit           20480  2 amdgpu,xe
drm_ttm_helper         20480  3 amdgpu,xe
ttm                   163840  3 amdgpu,drm_ttm_helper,xe
drm_exec               12288  3 drm_gpuvm,amdgpu,xe
gpu_sched              73728  2 amdgpu,xe
drm_suballoc_helper    24576  2 amdgpu,xe
drm_display_helper    364544  2 amdgpu,xe
video                  86016  2 amdgpu,xe
cec                   106496  3 drm_display_helper,amdgpu,xe

Actual Behavior

Disclosure: 26.18.38308.4 is the newest driver packaged for Fedora; the 26.27.39122.11 release ships only .deb packages, so I have not been able to test on it. I did scan the 26.18.38308.4 -> 26.27.39122.11 commit range (673 commits) by subject and found nothing that matches the path described below.

With llama.cpp, when allocating too much VRAM (such as when attempting to increase context size), the process detects the over-commit and then hangs in exit().

The process exit spins forever at 100% CPU. Backtrace:

#0 NEO::DrmDirectSubmission<Xe2HpgCoreFamily, RenderDispatcher<Xe2HpgCoreFamily>>::ensureRingCompletion()
#1 DirectSubmissionHw<Xe2HpgCoreFamily, RenderDispatcher<Xe2HpgCoreFamily>>::stopRingBuffer(bool)
#2 CommandStreamReceiverHw<Xe2HpgCoreFamily>::stopDirectSubmission(bool, bool)
#3 NEO::Device::stopDirectSubmissionAndWaitForCompletion()
#4 L0::Device::releaseResources()
#5 L0::Device::~Device()
#6 L0::DriverHandle::~DriverHandle()
#7 _dl_fini / exit()

No timeout, no error propagation. SIGKILL is required; the GPU recovers after the kill.

Expected Behavior

The function call should return and the application should exit cleanly.

Reproduction Rate

Always reproduces - 100%

Steps to Reproduce

l0-overcommit-repro.c

Compile and run the attached program.

Is this a regression?

  • Yes, this is a regression - functionality that previously worked is now broken

Last Known Working Driver Version

No response

First Known Failing Driver Version

No response

API Call Logs

No response

strace Logs

No response

System Logs / dmesg Output

$ sudo dmesg --since '4 hours ago' | grep -i -E 'i915|xe|drm|gpu' | tail -n 100
[70796.532842] traps: l0-overcommit-r[508307] general protection fault ip:7f0e9f17a7ed sp:7fff2af74c50 error:0 in libze_intel_gpu.so.1.15.38308[17a7ed,7f0e9f000000+7f4000]
[70813.099289] traps: l0-overcommit-r[508502] general protection fault ip:7fbc7157a7ed sp:7ffcb3efd250 error:0 in libze_intel_gpu.so.1.15.38308[17a7ed,7fbc71400000+7f4000]
[71030.507820] traps: l0-overcommit-r[510098] general protection fault ip:7fc32837a7ed sp:7ffc51150e70 error:0 in libze_intel_gpu.so.1.15.38308[17a7ed,7fc328200000+7f4000]
[71199.429479] traps: l0-overcommit-r[511201] general protection fault ip:7f0a8ff7a7ed sp:7ffd462b3420 error:0 in libze_intel_gpu.so.1.15.38308[17a7ed,7f0a8fe00000+7f4000]
[73746.587315] traps: l0-overcommit-r[534516] general protection fault ip:7f525f77a7ed sp:7ffe18c717a0 error:0 in libze_intel_gpu.so.1.15.38308[17a7ed,7f525f600000+7f4000]
[73775.132545] traps: l0-overcommit-r[534728] general protection fault ip:7fbc4257a7ed sp:7fff1ef315d0 error:0 in libze_intel_gpu.so.1.15.38308[17a7ed,7fbc42400000+7f4000]

Backtrace (if crash or hang occurred)

#0 NEO::DrmDirectSubmission<Xe2HpgCoreFamily, RenderDispatcher<Xe2HpgCoreFamily>>::ensureRingCompletion()
#1 DirectSubmissionHw<Xe2HpgCoreFamily, RenderDispatcher<Xe2HpgCoreFamily>>::stopRingBuffer(bool)
#2 CommandStreamReceiverHw<Xe2HpgCoreFamily>::stopDirectSubmission(bool, bool)
#3 NEO::Device::stopDirectSubmissionAndWaitForCompletion()
#4 L0::Device::releaseResources()
#5 L0::Device::~Device()
#6 L0::DriverHandle::~DriverHandle()
#7 _dl_fini / exit()

Source Code / Reproducer

l0-overcommit-repro.c

Compile and run the attached program.

Command Line / Application Details

No response

oneAPI Version (if applicable)

No response

Screenshots / Video

No response

Additional Notes

If this is related to #912 then this is a regression as I am on a later version of the driver.

Metadata

Metadata

Assignees

No one assigned

    Labels

    OS: LinuxIssue specific to Linux distributions (Ubuntu, Fedora, RHEL, etc.)Type: BugGeneral bug report, unexpected behavior or crash

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions