Persistent CUDA Error (device-side assert) during VITS Fine-tuning with Bengali Dataset on RTX 306

[surjoshikha]

Goal:
I am trying to fine-tune the tts_models/multilingual/multi-dataset/vits_vctk model on a custom Bengali (Bangla) voice dataset using Coqui TTS.

Dataset:

• LJSpeech format (filename|text|text).
• Approximately 422 .wav files (mono, 22050 Hz).
• Data has been verified (all files exist, are valid WAVs, not too short) and sanitized (unknown/hidden characters removed).
• A complete Bengali alphabet (including characters found in the dataset) is defined in config.json.

Configuration:

• Using the config.json provided in troubleshooting (includes custom alphabet, use_cuda: true, batch_size tested at 16, 8, and 4).
• restore_path: tts_models/multilingual/multi-dataset/vits_vctk

Persistent Error:
Despite extensive troubleshooting, training consistently crashes with RuntimeError: CUDA error: device-side assert triggered. The traceback consistently points to low-level CUDA operations within PyTorch, often torch.embedding or F.conv1d. This occurs even with very small batch sizes (e.g., 4).

Troubleshooting Steps Taken:

• Created a completely clean Python virtual environment (venv).
• Ensured correct metadata.txt format (3-column) and dataset path.
• Defined a complete custom Bengali character set in config.json.
• Used scripts to sanitize metadata.txt and remove unknown/hidden characters.
• Identified and temporarily removed outlier files (longest audio, longest text lines).
• Reduced batch_size significantly (down to 4).
• Attempted CPU training (use_cuda: false and CUDA_VISIBLE_DEVICES=""), but still encountered CUDA errors (indicating a deep conflict).
• Performed a clean install of the latest NVIDIA Studio Driver (581.57 for RTX 3060).
• Reinstalled PyTorch (ensuring CUDA 12.1 GPU version: torch-2.9.0+cu121) and torchcodec multiple times.
• Reinstalled Coqui TTS (pip install TTS).

System Environment:
OS:** Windows 11
GPU:** NVIDIA GeForce RTX 3060
NVIDIA Driver:** 581.57 (Studio Driver)
Python: Python 3.11.9 --version
PyTorch:Name: torch
Version: 2.5.1+cu121
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3-Clause
Location: K:\Speech_to_text\my_own_model\venv_new\Lib\site-packages
Requires: filelock, fsspec, jinja2, networkx, sympy, typing-extensions
Required-by: encodec, torchaudio, torchvision, trainer, TTS
TTS:Name: TTS
Version: 0.22.0
Summary: Deep learning for Text to Speech by Coqui.
Home-page: https://github.com/coqui-ai/TTS
Author: Eren Gölge
Author-email: egolge@coqui.ai
License: MPL-2.0
Location: K:\Speech_to_text\my_own_model\venv_new\Lib\site-packages
Requires: aiohttp, anyascii, bangla, bnnumerizer, bnunicodenormalizer, coqpit, cython, einops, encodec, flask, fsspec, g2pkk, gruut, hangul_romanize, inflect, jamo, jieba, librosa, matplotlib, nltk, num2words, numba, numpy, packaging, pandas, pypinyin, pysbd, pyyaml, scikit-learn, scipy, soundfile, spacy, torch, torchaudio, tqdm, trainer, transformers, umap-learn, unidecode
Required-by:

Request:
Could this persistent CUDA error be related to a known incompatibility with the RTX 3060, the specific driver version, or a subtle issue in the VITS model's handling of Bengali script during fine-tuning? Any insights or suggestions would be greatly appreciated.

config.json
{
"run_name": "MyPerfectBengaliModel_V1",
"output_path": "training_output/",
"model": "vits",
"phonemizer": {},
"use_phonemes": false,
"characters": {
"pad": "_",
"eos": "~",
"bos": "^",
"characters": "অআইঈউঊঋএঐওঔকখগঘঙচছজঝঞটঠডঢণতথদধনপফবভমযরলশষসহড়ঢ়য়ৎংঃঁািীুূৃেৈোৌ্ABCDFIMNSTacdefghilmnorstu۱۲৩৪৫৬৭৮৯০0123468*‘॥‌–—’\ufeffড়য়১২",
"punctuations": "।,-;? !'"().:",
"phonemes": ""
},
"model_args": {
"num_speakers": 0,
"use_d_vector_file": false,
"d_vector_file": null,
"d_vector_dim": 0
},
"datasets": [
{
"formatter": "ljspeech",
"name": "FinalFormattedDataset",
"path": "FinalFormattedDataset/",
"meta_file_train": "metadata.txt"
}
],
"eval_split_size": 0.02,
"restore_path": "tts_models/multilingual/multi-dataset/vits_vctk",
"batch_size": 8,
"eval_batch_size": 8,
"epochs": 1000,
"learning_rate": 0.0001,
"num_loader_workers": 4,
"num_eval_loader_workers": 4,
"save_step": 1000,
"print_step": 25,
"use_cuda": true
}

[eginhard]

This is likely the same issue as idiap#424, set your character config as described there, just replacing your characters with the Bengali ones.

I'd also strongly recommend to use our fork (available via pip install coqui-tts). I think there it should be fixed, along with many other bugs plus it works with recent versions of Python and Pytorch. This repo is not maintained anymore.