docs: add VRAM requirements + T4/Turing hardware notes#200
Open
moduvoice wants to merge 2 commits into
Open
Conversation
Add measured VRAM figures for the 8B flagship (MOSS-TTS-v1.5, OOMs on 16GB GPUs) vs. the 4B MOSS-TTS-Local-Transformer-v1.5 (fits comfortably on 16GB GPUs), a torchcodec/LD_LIBRARY_PATH troubleshooting note for the torch-runtime extra, a dtype hint for Turing GPUs (e.g. T4), and a note that bitsandbytes int8 quantization does not reduce VRAM usage on this architecture. No code changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
While validating MOSS-TTS on an NVIDIA T4 (16 GB, Turing/sm_75), I hit several undocumented hardware/dependency gotchas that aren't logic bugs but cost real time to diagnose:
OpenMOSS-Team/MOSS-TTS-v1.5,MossTTSDelay) in bf16 with.to("cuda"). That alone needs ≈15.5 GB of VRAM (measured:torch.OutOfMemoryErrorwith "15.50 GiB in use" of a 15.56 GiB total), leaving no room for activations/KV-cache — it OOMs on any 16 GB GPU including the T4. Neither the README nor the model card mention this.[torch-runtime]extra withuv/pip, then callingtorchaudio.save()on the 4BMOSS-TTS-Local-Transformer-v1.5checkpoint raisesRuntimeError: Could not load libtorchcodec, even with FFmpeg correctly installed. The actual cause is (a) the pip-installed torch/nvidia CUDA shared libraries aren't onLD_LIBRARY_PATH, and (b)nvidia-npp-cu12is a transitive dependency oftorchcodecthat isn't declared/installed automatically.moss_tts_local_v1.5/README.mdsetdtype = torch.bfloat16 if device == "cuda" else torch.float32with no GPU-generation check. On Turing (T4, compute capability 7.5), bf16 inputs are rejected by PyTorch's SDPAEFFICIENT_ATTENTIONkernel (RuntimeError: No available kernel), silently forcing a fallback to the slower MATH backend.torch.float16avoids this.quantization_config=BitsAndBytesConfig(load_in_8bit=True)loads without error but gives no measurable VRAM reduction on this architecture (measured on both the 4B and 8B checkpoints), so it isn't a viable workaround for fitting the 8B model on a smaller GPU.By contrast, the repo's
resolve_attn_implementation()already checkstorch.cuda.get_device_capability()[0] >= 8before choosingflash_attention_2, so it correctly falls back tosdpaon the T4 without a crash — this PR doesn't touch that logic, only adds docs around the surrounding gaps.Changes
Doc-only, no code changes:
README.md: VRAM note after the "Released Models" table pointing 16 GB-GPU users toMOSS-TTS-Local-Transformer-v1.5(4B) orMOSS-TTS-Local-Transformer(1.7B) instead of the 8B flagship, plus a note that bitsandbytes int8 doesn't reduce VRAM here.README.md: troubleshooting note under "Environment Setup" for theCould not load libtorchcodecerror (LD_LIBRARY_PATH+nvidia-npp-cu12).README.mdandmoss_tts_local_v1.5/README.md: inline comment next to thedtype = torch.bfloat16 if device == "cuda" ...line noting that Turing GPUs (e.g. T4) should usetorch.float16instead.moss_tts_local_v1.5/README.md: VRAM note confirming this 4B checkpoint fits a 16 GB GPU.Testing
Measured on an NVIDIA T4 (16 GB, driver 550.163.01, torch 2.9.1+cu128, transformers 5.0.0):
MOSS-TTS-v1.5) bf16 load: OOM, "15.50 GiB in use" of 15.56 GiB total (with and withoutload_in_8bit=True— same crash point, quantization not applied before the OOM).MOSS-TTS-Local-Transformer-v1.5): peak VRAM ≈13.4 GB (bf16 and fp16), fits with ~16.5% headroom.EFFICIENT_ATTENTION→RuntimeError: No available kernel; fp16/fp32 +EFFICIENT_ATTENTION→ OK; bf16 +MATH→ OK.libtorchcodecload failure reproduced and resolved step-by-step viaLD_LIBRARY_PATH(libtorch.so, thenlibnvrtc.so.12) andpip install nvidia-npp-cu12(libnppicc.so.12).No code paths were changed; only documentation/comments were added.