kernel: clamp zero-size host registrations; WDDM lock-pool hint on pin failure

[Glucksberg]

What

Two small, self-contained fixes found while debugging Windows/WDDM pin-quota failures with large MoE models (full repro and measurements in #55):

1. host_register: clamp zero-byte registrations to 1 byte. cudaHostRegister rejects a 0-byte request with cudaErrorInvalidValue. The NVFP4/FP8 offload loaders register per-layer scale banks whose padded size can round to zero on dense layers, which aborts the entire pin pipeline with a cryptic cudaHostRegister failed for 0.0 GiB. alloc_pinned_tensor in the same file already clamps (nbytes == 0 ? 1 : nbytes) — this makes host_register consistent with it.

2. HostBank.pin: actionable error on lock-pool exhaustion. On Windows/WDDM the pageable-locking pool is roughly half of system RAM, so registering ~17 GB of MoE banks on a 32 GB box fails with a bare driver OOM long before system RAM is exhausted. The re-raised error now carries the bank size plus a hint (free RAM / add RAM / --moe-backend cpu), instead of just cudaHostRegister failed for 13.77 GiB.

Validation

• Repro machine: RTX 3080 Ti 12 GB / 32 GB RAM / driver 591.86 (CUDA 13.1), serving ornith-ai/Ornith-1.5-35B-A3B-NVFP4.
• Before: serve aborts at PinPipeline.wait() with RuntimeError: cudaHostRegister failed for 0.0 GiB.
• With an equivalent runtime patch (clamp + graceful handling), registration proceeds normally until the WDDM pool ceiling (~13.8 GB measured) — see Windows/WDDM: offload serving fails when bank sources exceed the ~50%-RAM pageable-locking pool (repro, measurements, proposed path) #55 for the instrumented numbers.
• The clamp itself is exercised by the same path (alloc_pinned_tensor has used the identical guard since it shipped).

Notes

These unblock diagnosing the bigger residency gap tracked in #55 (LOCKED/PAGEABLE movement paths are scaffolded but not implemented). Not included here to keep the diff minimal.

[gdevenyi]

Merged and tested on Linux / 2× RTX 6000 Ada, serving a 143 GiB offloaded expert pool. No regressions — full suite 1606 passed, and nothing attributable to this change.

The zero-size clamp is worth having on Linux too, not just WDDM: cudaHostRegister with a zero length is an easy edge to hit from an empty bank or a model with a degenerate layer, and turning it into a no-op rather than an error is clearly right.

On the pin-failure guidance: I hit the Linux flavour of this problem from the other direction and it cost about an hour, so the instinct behind this PR is a good one. There, cudaHostRegister failed partway through registering ~155 GiB because RLIMIT_MEMLOCK defaults to RAM/8 (62 GiB on a 503 GiB box), and the failure surfaced minutes into a load with no hint about memlock. If you are already adding platform-specific guidance to the failure path, a Linux branch pointing at ulimit -l / /etc/security/limits.d/ would land in exactly the same place and save the same hour.

Worth knowing if you add that: limits apply at session start, so an SSH ControlMaster will keep serving the old limit after the config is fixed, which makes the fix look like it did not work. ssh -O exit <host> first.

+1 as is.

[Glucksberg]

Thanks @gdevenyi — pushed 815c673 to this branch adding the Linux branch you asked for: the re-raised error now reports the current RLIMIT_MEMLOCK soft limit and points at ulimit -l / /etc/security/limits.d/*.conf, including the ControlMaster gotcha (ssh -O exit before re-testing). Windows branch unchanged. Ready for another look whenever.