Skip to content

docs: add VRAM requirements + T4/Turing hardware notes#200

Open
moduvoice wants to merge 2 commits into
OpenMOSS:mainfrom
moduvoice:docs/t4-turing-notes
Open

docs: add VRAM requirements + T4/Turing hardware notes#200
moduvoice wants to merge 2 commits into
OpenMOSS:mainfrom
moduvoice:docs/t4-turing-notes

Conversation

@moduvoice

Copy link
Copy Markdown

Motivation

While validating MOSS-TTS on an NVIDIA T4 (16 GB, Turing/sm_75), I hit several undocumented hardware/dependency gotchas that aren't logic bugs but cost real time to diagnose:

  1. The README's headline "MOSS‑TTS Basic Usage" example loads the flagship 8B checkpoint (OpenMOSS-Team/MOSS-TTS-v1.5, MossTTSDelay) in bf16 with .to("cuda"). That alone needs ≈15.5 GB of VRAM (measured: torch.OutOfMemoryError with "15.50 GiB in use" of a 15.56 GiB total), leaving no room for activations/KV-cache — it OOMs on any 16 GB GPU including the T4. Neither the README nor the model card mention this.
  2. Installing the documented [torch-runtime] extra with uv/pip, then calling torchaudio.save() on the 4B MOSS-TTS-Local-Transformer-v1.5 checkpoint raises RuntimeError: Could not load libtorchcodec, even with FFmpeg correctly installed. The actual cause is (a) the pip-installed torch/nvidia CUDA shared libraries aren't on LD_LIBRARY_PATH, and (b) nvidia-npp-cu12 is a transitive dependency of torchcodec that isn't declared/installed automatically.
  3. Both the main README and moss_tts_local_v1.5/README.md set dtype = torch.bfloat16 if device == "cuda" else torch.float32 with no GPU-generation check. On Turing (T4, compute capability 7.5), bf16 inputs are rejected by PyTorch's SDPA EFFICIENT_ATTENTION kernel (RuntimeError: No available kernel), silently forcing a fallback to the slower MATH backend. torch.float16 avoids this.
  4. Passing quantization_config=BitsAndBytesConfig(load_in_8bit=True) loads without error but gives no measurable VRAM reduction on this architecture (measured on both the 4B and 8B checkpoints), so it isn't a viable workaround for fitting the 8B model on a smaller GPU.

By contrast, the repo's resolve_attn_implementation() already checks torch.cuda.get_device_capability()[0] >= 8 before choosing flash_attention_2, so it correctly falls back to sdpa on the T4 without a crash — this PR doesn't touch that logic, only adds docs around the surrounding gaps.

Changes

Doc-only, no code changes:

  • README.md: VRAM note after the "Released Models" table pointing 16 GB-GPU users to MOSS-TTS-Local-Transformer-v1.5 (4B) or MOSS-TTS-Local-Transformer (1.7B) instead of the 8B flagship, plus a note that bitsandbytes int8 doesn't reduce VRAM here.
  • README.md: troubleshooting note under "Environment Setup" for the Could not load libtorchcodec error (LD_LIBRARY_PATH + nvidia-npp-cu12).
  • README.md and moss_tts_local_v1.5/README.md: inline comment next to the dtype = torch.bfloat16 if device == "cuda" ... line noting that Turing GPUs (e.g. T4) should use torch.float16 instead.
  • moss_tts_local_v1.5/README.md: VRAM note confirming this 4B checkpoint fits a 16 GB GPU.

Testing

Measured on an NVIDIA T4 (16 GB, driver 550.163.01, torch 2.9.1+cu128, transformers 5.0.0):

  • 8B (MOSS-TTS-v1.5) bf16 load: OOM, "15.50 GiB in use" of 15.56 GiB total (with and without load_in_8bit=True — same crash point, quantization not applied before the OOM).
  • 4B (MOSS-TTS-Local-Transformer-v1.5): peak VRAM ≈13.4 GB (bf16 and fp16), fits with ~16.5% headroom.
  • 4B int8 (bitsandbytes): peak VRAM ≈12.5 GB — essentially the same as (slightly higher than) bf16's ≈12.0 GB at load time, i.e. no reduction.
  • Synthetic SDPA backend check: bf16 + EFFICIENT_ATTENTIONRuntimeError: No available kernel; fp16/fp32 + EFFICIENT_ATTENTION → OK; bf16 + MATH → OK.
  • libtorchcodec load failure reproduced and resolved step-by-step via LD_LIBRARY_PATH (libtorch.so, then libnvrtc.so.12) and pip install nvidia-npp-cu12 (libnppicc.so.12).

No code paths were changed; only documentation/comments were added.

Add measured VRAM figures for the 8B flagship (MOSS-TTS-v1.5, OOMs on
16GB GPUs) vs. the 4B MOSS-TTS-Local-Transformer-v1.5 (fits comfortably
on 16GB GPUs), a torchcodec/LD_LIBRARY_PATH troubleshooting note for
the torch-runtime extra, a dtype hint for Turing GPUs (e.g. T4), and a
note that bitsandbytes int8 quantization does not reduce VRAM usage on
this architecture. No code changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant