CUDA: Support CUDA Virtual Devices

[anavp-nvidia]

Overview

Adds support for exposing a configurable number of virtual CUDA devices on top of the physical GPUs actually present, controlled by a new GGML_CUDA_DEVICES environment variable. This allows developing and exercising multi-GPU code paths on machines with a single GPU.

When GGML_CUDA_DEVICES is unset, behavior is identical to before.

-sm layer, -sm row, and -sm tensor are all supported under virtual CUDA devices. For -sm tensor, the internal CUDA AllReduce or the meta-backend fallbacks are supported since NCCL seems to require distinct physical GPUs.

Additional information

The core idea is a single indirection: every (virtual) device id is mapped to a backing physical device id, and all raw CUDA calls that take a device ordinal use the physical id, while ggml-level bookkeeping (pools, buffers, device registry) stays per-virtual-device.

Design choices

• ggml_cuda_init() parses GGML_CUDA_DEVICES, builds the round-robin virtual to physical map, and computes how many virtual devices share each GPU. ggml_cuda_set_device() resolves to the physical device, and a new ggml_cuda_get_physical_device() exposes the mapping.
• ggml_cuda_device_info gains physical_device_count and, per device, physical_device and physical_share_count.
• Property/attribute/VMM/peer-copy/PCI/memInfo calls use the physical id. Device registration appends a -v<i> suffix to each virtual device's pci_bus_id so they stay unique.
• op_mul_mat_device (context member, set by ggml_cuda_op_mul_mat dispatcher) is used to identify the main device in mul mat execution instead of ggml_cuda_get_device(), which returns the physical device and was otherwise causing out-of-bounds writes.
• VRAM accounting: a virtual device reports total/physical_share_count for both free and total memory, so per-device VRAM sums to the physical total and model distribution doesn't overcommit a shared GPU.

Testing

Validated on three Blackwell systems, using four models (Llama-3.2-1B-Instruct, Llama-3.1-8B-Instruct, Qwen2.5-32B-Instruct, Llama-3.3-70B-Instruct) with greedy decoding (temp 0) and a fixed seed, so as to compare virtual-device output against physical-GPU reference.

System (GPUs)	VRAM	Coverage
2× B200 (Linux)	2 × 192 GB	op + e2e + VRAM tests
2× RTX 5090 (Windows)	2 × 32 GB	e2e tests
1× RTX PRO 6000 Blackwell (Windows)	1 × 96 GB	e2e tests + single-GPU emulation

• Tested ops using test-backend-ops under 2 vdev (virtual device) / 1 phys (physical device) configuration. Tested MUL_MAT / MUL_MAT_ID on baseline, 2 vdev / 1 phys and 4 vdev / 2 phys configurations.
• Ran end-to-end tests across configurations including 2 vdev / 2 phys, 2 vdev / 1 phys, 3 vdev / 2 phys, 4 vdev / 2 phys, 4 vdev / 1 phys for -sm layer, -sm row, and -sm tensor (non-NCCL paths):
  • -sm layer / -sm row: output identical to the real-GPU baseline (flash-attn on/off on 1B/8B; on for 32B/70B)
  • -sm tensor: valid coherent output, deterministic across configurations (n vdev / 1 phys where n = 2, 3, 4), and independent of the virtual to physical mapping (n vdev / 1 phys produces the same as n vdev / 2 phys, n = 2, 4). Both all-reduce fallbacks exercised (GGML_CUDA_ALLREDUCE=internal/none).
• VRAM accounting: per-virtual-device memory = physical / N and sums back to the physical total.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES, AI was used to understand the code base, for the initial draft of changes, for overall validation and thorough testing.

[gaugarg-nv]

Could you please rebase your changes on the TOT master?

[Kononnable]

This allows developing and exercising multi-GPU code paths on machines with a single GPU.

Can't this already be achieved by using RPC without any additional code changes? Connecting to a ggml-rpc-server running on the same machine as llama-server should result in multiple devices being recognized and should allow for multi-gpu code paths on a machine with a single gpu to be used for testing/development.

[gaugarg-nv]

IIUC, these dev_ids are virtual devices, ncclCommInitAll will fail if you pass [0,1] on a single device.

I think it might be better to disable the NCCL path for the virtual device case.

[gaugarg-nv]

Does the internal allReduce implementation work fine with virtual devices?