595.71.05: VA-space exhaustion in reuse-mapping DB wedges a channel; RC recovery never completes; process becomes unkillable

[truebest]

nvidia-bug-report.log.gz

System

• GPU: NVIDIA GeForce RTX 5060 Ti (GB206, PCI 10de:2d04 rev a1)
• Driver: 595.71.05, open kernel modules, CUDA 13.2
• Kernel: 7.0.0-28-generic, Ubuntu 26.04 LTS, Wayland (GNOME) desktop on the same GPU
• Workload: gnome-remote-desktop 50.0 (RDP server) encoding AVC420 via its CUDA/NVENC
  hardware path, streaming continuously to one client

Summary

After ~27 hours of continuous RDP streaming, the driver's reuse-mapping DB fails VA
allocation (mapping_reuse.c:273, NV_ERR_NO_MEMORY) while device VRAM is nearly
empty (1.5 GiB / 16 GiB). The channel that hit the failure wedges, RC recovery never
completes (krcWatchdog_IMPL: RC watchdog: GPU is probably locked! repeating every
~8 s, 150+ occurrences and counting), and the owning process can no longer be killed:
SIGKILL is delivered but a thread never returns from the driver ioctl, so the device
stays open, which in turn makes nvidia-smi --gpu-reset / module reload impossible.
Only a full reboot recovers. Display output on other channels keeps working the whole
time.

Two independent defects from one incident:

1. A VA/mapping leak (or unbounded reuse-mapping DB growth) under long-running
   map/unmap-heavy CUDA interop — VRAM stays low while VA space for mappings runs out.
2. RC recovery livelock: the watchdog detects the locked channel every 8 s but never
   recovers it, and the stuck ioctl makes the process unkillable, so there is no
   userspace path back to a working GPU.

Timeline (kernel + service logs)

• Jul 31 06:27:08 — gnome-remote-desktop starts; [HWAccel.CUDA] Initialization of CUDA was successful; streaming works for ~27 h.
• Aug 01 09:11:25 — kernel: repeated
  NVRM: dmaAllocMapping_GM107: can't alloc VA space for mapping. (22 total) and
  NVRM: nvAssertOkFailedNoLog: Assertion failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from pReuseMappingDb->pMapCb(...) @ mapping_reuse.c:273
• Aug 01 09:11:36 — kernel: first
  NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked! Notify Timeout Seconds: 7 — repeats every ~8 s from then on (154 at the time of writing).
• Aug 01 09:11:47 — gnome-remote-desktop: [HWAccel.CUDA] Failed to map resources: the launch timed out and was terminated; from here every new RDP connection fails
  with Failed to create stream: the launch timed out and was terminated →
  Layout manager: Failed to prepare surface contexts → session closed with
  ERRINFO_CLOSE_STACK_ON_DRIVER_FAILURE.
• Aug 01 09:19 — systemctl --user restart gnome-remote-desktop hangs in
  deactivating; explicit SIGKILL delivered, process (PID 4003622) never exits;
  no D-state threads reported, but the task never leaves the driver.
• GPU state throughout: display fine, nvidia-smi responsive, P1, 0% util,
  1566 MiB / 16311 MiB used; no Xid events logged at any point.

Expected behavior

• The mapping DB should not exhaust VA space under a steady map/unmap workload
  (or should fail gracefully long before wedging a channel).
• RC recovery should tear down the faulted channel so the process can exit and the
  device can be reset without rebooting.

Reproduction

Long-running gnome-remote-desktop RDP session with the CUDA/NVENC path on Blackwell +
open modules; failure appeared after ~27 h of continuous streaming. Not yet re-run to
confirm the period.

Attachments

• nvidia-bug-report.log.gz captured while the GPU was in the wedged state will be
  attached in a follow-up comment (one shell working-directory path redacted from a
  sudo journal line; nothing else modified).
• Full journal excerpts available on request.

[truebest]

I’ll drop the file later, because session is dead and agent wasn’t able to upload this file via gh, and I bit away from actual computer

[truebest]

Additional observation while the GPU is in this state: creating a NEW GPU context hangs indefinitely. glxinfo -B (new GL context on the NVIDIA device) blocks with no output and had to be killed after 10 s; no new NVRM errors are logged while it hangs. Existing contexts (Wayland compositor, running apps) keep rendering normally the whole time. So the locked channel + non-completing RC recovery not only pins the faulted process, it also blocks channel/context creation for new clients driver-wide.