[Bug / Performance] 长音频 preset_spk_num 绕过 large-N 防护并触发极高开销的谱聚类 / Long audio with preset_spk_num bypasses large-N clustering safeguard

## 🐛 Bug

在使用 FunASR 的 speaker diarization 时，如果对长音频通过 `preset_spk_num` 指定明确的说话人数，会导致 `ClusterBackend` 绕过现有的 large-N 聚类保护逻辑，强制进入 Spectral Clustering。

对于本次约 **2 小时 16 分钟**的音频，最终产生：

```text
X.shape=torch.Size([10639, 192]), oracle_num=2
```

此时 FunASR 会对超过一万条 speaker embeddings 构造 dense affinity / Laplacian matrix，并在：

```python
scipy.linalg.eigh()
```

中执行 dense eigendecomposition。

实际表现为：

* CPU 长时间满载；
* worker 长时间无法释放；
* 在队列式 ASR 服务中导致后续任务被阻塞；
* 看起来类似“任务卡死”，但 profiler 显示进程实际上一直在 `scipy.linalg.eigh()` 中执行计算。

相同音频仅取消 `preset_spk_num=2` 后：

```text
X.shape=torch.Size([10639, 192]), oracle_num=None
```

即可正常完成，不再出现异常的 CPU 长时间占用。

---

When using FunASR speaker diarization on long audio, specifying a known speaker count through `preset_spk_num` causes `ClusterBackend` to bypass the existing large-N clustering safeguard and forces Spectral Clustering.

For the reproduced audio, approximately **2h 16m 34s** long, the speaker clustering input becomes:

```text
X.shape=torch.Size([10639, 192]), oracle_num=2
```

FunASR then constructs dense affinity / Laplacian matrices for more than ten thousand speaker embeddings and performs a dense eigendecomposition through:

```python
scipy.linalg.eigh()
```

In production this causes:

* sustained CPU saturation;
* the worker remaining occupied for a very long time;
* head-of-line blocking for subsequent ASR jobs;
* behavior that appears to be a hung task, while profiling shows that the process is actively spending CPU time in `scipy.linalg.eigh()`.

Using the exact same audio without `preset_spk_num=2` results in:

```text
X.shape=torch.Size([10639, 192]), oracle_num=None
```

and the abnormal prolonged CPU utilization disappears.

---

## To Reproduce

1. Install FunASR and the dependencies required for CAM++ speaker diarization.

2. Initialize an `AutoModel` with speaker diarization enabled:

```python
from funasr import AutoModel

model = AutoModel(
    model="paraformer-zh",
    vad_model="fsmn-vad",
    vad_kwargs={"max_single_segment_time": 30000},
    punc_model="ct-punc-c",
    spk_model="cam++",
    device="cuda:0",
    disable_update=True,
)
```

3. Run speaker diarization on a long audio file while specifying a fixed speaker count:

```python
result = model.generate(
    input=audio_filepath,
    cache={},
    language="zh",
    disable_pbar=True,
    batch_size_s=120,
    use_itn=True,
    merge_vad=True,
    merge_length_s=15,
    preset_spk_num=2,
)
```

4. Internally the parameter propagation is:

```text
preset_spk_num=2
→ ClusterBackend(..., oracle_num=2)
```

5. For the reproduced audio, temporary logging added to `ClusterBackend.forward()` shows:

```text
[SpeakerCluster] X.shape=torch.Size([10639, 192]), oracle_num=2
```

6. Observe the running process:

```bash
sudo py-spy top --pid <PID>
```

7. The process spends essentially all sampled CPU time inside:

```text
scipy.linalg.eigh
```

8. Run the exact same audio again without specifying `preset_spk_num`:

```python
result = model.generate(
    input=audio_filepath,
    cache={},
    language="zh",
    disable_pbar=True,
    batch_size_s=120,
    use_itn=True,
    merge_vad=True,
    merge_length_s=15,
)
```

The clustering log becomes:

```text
[SpeakerCluster] X.shape=torch.Size([10639, 192]), oracle_num=None
```

and the abnormal prolonged CPU usage disappears.

### Relevant FunASR code path

Current logic in:

```text
funasr/models/campplus/cluster_backend.py
```

is:

```python
k = params["oracle_num"] if "oracle_num" in params else None

assert len(X.shape) == 2, "modelscope error: the shape of input should be [N, C]"

if X.shape[0] < 20:
    return np.zeros(X.shape[0], dtype="int")

if X.shape[0] < 2048 or k is not None:
    labels = self.spectral_cluster(X, k)
else:
    labels = self.umap_hdbscan_cluster(X)
```

The problematic condition appears to be:

```python
or k is not None
```

When `oracle_num` is supplied, arbitrarily large `X.shape[0]` values can enter the Spectral Clustering path, bypassing the existing `2048` large-N threshold.

The spectral clustering implementation later executes:

```python
lambdas, eig_vecs = scipy.linalg.eigh(L)
```

For this reproduction:

```text
N = 10639
```

so the dense Laplacian is approximately:

```text
10639 × 10639
```

containing:

```text
113,188,321
```

matrix elements.

A full dense symmetric eigendecomposition at this scale is extremely expensive.

---

## Code sample

Minimal reproducing usage:

```python
from funasr import AutoModel

model = AutoModel(
    model="paraformer-zh",
    vad_model="fsmn-vad",
    vad_kwargs={"max_single_segment_time": 30000},
    punc_model="ct-punc-c",
    spk_model="cam++",
    device="cuda:0",
    disable_update=True,
)

result = model.generate(
    input=audio_filepath,
    cache={},
    language="zh",
    disable_pbar=True,
    batch_size_s=120,
    use_itn=True,
    merge_vad=True,
    merge_length_s=15,
    preset_spk_num=2,
)
```

For comparison, removing only:

```python
preset_spk_num=2
```

from the same invocation avoids the problematic clustering path.

In the actual application, the external API parameter is called:

```text
speaker_count=2
```

and is mapped as:

```text
speaker_count=2
→ model.generate(preset_spk_num=2)
→ ClusterBackend(..., oracle_num=2)
```

This mapping is application-side only; the FunASR API parameter involved in the reproduction is `preset_spk_num`.

---

## Expected behavior

指定已知说话人数不应该导致 large-N 的保护逻辑失效。

从 API 使用者角度：

```python
preset_spk_num=2
```

是合法且合理的输入，不应该导致算法在 large-N 情况下从可扩展聚类路径切换到对超过一万条 embedding 执行 dense Spectral Clustering。

合理的调度策略可以类似：

```text
small N
    → Spectral Clustering

large N + unknown speaker count
    → UMAP + HDBSCAN

large N + known speaker count
    → scalable fixed-K clustering
```

例如 large-N + known-K 可以考虑：

```text
L2-normalized speaker embeddings
→ KMeans(K=oracle_num)
```

或者：

```text
UMAP
→ KMeans(K=oracle_num)
```

具体算法由维护者评估即可。

至少应保证：

1. large-N + `oracle_num` 不会无条件进入 full dense `scipy.linalg.eigh()`；
2. `oracle_num` 不会静默绕过 large-N safeguard；
3. 如果当前没有适合的 fixed-K large-N backend，应至少给出 warning 或安全 fallback；
4. 合法的长音频请求不应导致 worker 因聚类阶段长时间不可用。

---

Providing a known speaker count should not disable the large-N scalability safeguard.

From an API user's perspective:

```python
preset_spk_num=2
```

is a valid and reasonable input and should not cause the clustering algorithm to switch from a scalable large-N path to dense Spectral Clustering over more than ten thousand speaker embeddings.

A possible dispatch strategy could be:

```text
small N
    → Spectral Clustering

large N + unknown speaker count
    → UMAP + HDBSCAN

large N + known speaker count
    → scalable fixed-K clustering
```

For example:

```text
L2-normalized speaker embeddings
→ KMeans(K=oracle_num)
```

or:

```text
UMAP
→ KMeans(K=oracle_num)
```

The exact clustering algorithm can of course be determined by the maintainers.

At minimum:

1. large-N + `oracle_num` should not unconditionally enter full dense `scipy.linalg.eigh()`;
2. `oracle_num` should not silently bypass the large-N safeguard;
3. if no scalable fixed-K backend is currently available, a warning or safe fallback should be provided;
4. a valid long-audio request should not keep a worker occupied for an effectively unbounded amount of time.

---

## Error logs

There is no Python exception or traceback.

The task remains active but becomes CPU-bound for a very long time.

`py-spy` output:

```text
11.59s behind in sampling, results may be inaccurate.
Try reducing the sampling rate.

Collecting samples from 'python main.py' (python v3.14.6)

Total Samples 445
GIL: 100.00%, Active: 100.00%, Threads: 5

  %Own   %Total  OwnTime  TotalTime  Function (filename)

100.00% 100.00%    4.45s     4.45s   eigh (scipy/linalg/_decomp.py)
  0.00% 100.00%   0.000s     4.45s   transcribe_file (asr_worker.py)
  0.00% 100.00%   0.000s     4.45s   __call__ (funasr/models/campplus/cluster_backend.py)
  0.00% 100.00%   0.000s     4.45s   _call_impl (torch/nn/modules/module.py)
  0.00% 100.00%   0.000s     4.45s   run (asr_worker.py)
  0.00% 100.00%   0.000s     4.45s   forward (funasr/models/campplus/cluster_backend.py)
  0.00% 100.00%   0.000s     4.45s   _generate (asr_worker.py)
  0.00% 100.00%   0.000s     4.45s   inference_with_vad (funasr/auto/auto_model.py)
  0.00% 100.00%   0.000s     4.45s   wrapper (scipy/_lib/_util.py)
  0.00% 100.00%   0.000s     4.45s   get_spec_embs (funasr/models/campplus/cluster_backend.py)
  0.00% 100.00%   0.000s     4.45s   _wrapped_call_impl (torch/nn/modules/module.py)
  0.00% 100.00%   0.000s     4.45s   generate (funasr/auto/auto_model.py)
```

Additional logging:

```text
[SpeakerCluster] X.shape=torch.Size([10639, 192]), oracle_num=2
```

A/B test with the fixed speaker count removed:

```text
[SpeakerCluster] X.shape=torch.Size([10639, 192]), oracle_num=None
```

After this change, the abnormal prolonged CPU utilization disappears.

There is also an unrelated ROCm/MIOpen warning during execution:

```text
MIOpen(HIP): Warning [IsEnoughWorkspace] [EvaluateInvokers]
Solver <GemmFwdRest>, workspace required: 113664,
provided ptr: 0 size: 0
```

However, this does not appear to be related to the reported issue because profiling clearly shows that the prolonged CPU-bound stage is inside:

```text
funasr/models/campplus/cluster_backend.py
→ get_spec_embs()
→ scipy.linalg.eigh()
```

---

## Environment

* OS: Debian 13, Linux `7.1.3+deb13-amd64`, glibc `2.41`
* Python version: `3.14.6`
* FunASR version: `1.4.2`
* ModelScope version: `1.39.1`
* PyTorch version: `2.13.0+rocm7.2`
* torchaudio version: `2.11.0+rocm7.2`
* SciPy version: `1.18.0`
* scikit-learn version: `1.9.0`
* NumPy version: `2.5.2`
* umap-learn version: `0.5.12`
* Install method: `pip`, Python virtual environment
* Device: `ROCm / HIP`
* GPU model: `AMD Radeon 8060S Graphics`
* GPU architecture: `gfx1151`
* ROCm / HIP version reported by PyTorch: `7.2.53211`
* `torch.cuda.is_available()`: `True`
* `torch.version.cuda`: `None`
* `torch.version.hip`: `7.2.53211`
* PyTorch device capability: `(11, 5)`
* Docker: Not used

---

## Audio details

* Duration: `8193.96 s` ≈ `2h 16m 33.96s`
* Format: `MP3`
* Codec: `MP3`
* Sample rate: `16000 Hz`
* Channels: `2`
* Channel layout: `stereo`
* Bit rate: `16000 bit/s`
* File size: `16,387,920 bytes`
* Language/dialect: Chinese / 中文
* Speaker count: `2`
* Background noise/music: Unknown / not relevant to reproducing the clustering behavior
* Speaker overlap: Unknown / not relevant to reproducing the clustering behavior
* Speaker embedding count after VAD / diarization preprocessing: `10639`
* Speaker embedding dimension: `192`

---

## Additional investigation / 补充分析

The current `ClusterBackend` contains:

```python
if X.shape[0] < 2048 or k is not None:
    labels = self.spectral_cluster(X, k)
else:
    labels = self.umap_hdbscan_cluster(X)
```

The explicit:

```python
X.shape[0] < 2048
```

condition suggests that scalability limitations of Spectral Clustering are already considered in the implementation.

However:

```python
or k is not None
```

allows a known speaker count to bypass this safeguard completely.

The A/B reproduction isolates this branch quite clearly:

```text
N = 10639
oracle_num = 2

→ Spectral Clustering
→ dense 10639 × 10639 Laplacian
→ scipy.linalg.eigh()
→ prolonged CPU saturation
```

versus:

```text
N = 10639
oracle_num = None

→ UMAP + HDBSCAN
→ normal completion
```

I also searched existing FunASR issues before submitting.

Related issues such as #1302 and #1372 involve long-audio speaker diarization but appear to have different root causes:

* #1302: CUDA Tensor → NumPy conversion issue;
* #1372: UMAP / pynndescent / NumPy error on long audio.

I could not find an existing issue specifically covering:

```text
preset_spk_num
→ oracle_num
→ bypass large-N threshold
→ Spectral Clustering
→ scipy.linalg.eigh()
→ extreme CPU cost
```

所以目前判断本 issue 与已有问题并不重复。

---

## Possible fix direction / 可能的修复方向

One possible dispatch structure could be:

```python
if n < 20:
    ...

elif n < LARGE_N_THRESHOLD:
    labels = spectral_cluster(X, k)

elif k is None:
    labels = umap_hdbscan_cluster(X)

else:
    labels = scalable_fixed_k_cluster(X, k)
```

The important point is not necessarily which clustering algorithm is chosen, but that:

```text
large N + known K
```

has a bounded/scalable execution path instead of unconditionally invoking dense Spectral Clustering.

Thanks for maintaining FunASR.

翻译使用了LLM进行整理 如有任何不标准的地方十分抱歉 可以评论我会及时补充

Navigation Menu

Uh oh!

[Bug / Performance] 长音频 preset_spk_num 绕过 large-N 防护并触发极高开销的谱聚类 / Long audio with preset_spk_num bypasses large-N clustering safeguard #3514

🐛 Bug

To Reproduce

Relevant FunASR code path

Code sample

Expected behavior

Error logs

Environment

Audio details

Additional investigation / 补充分析

Possible fix direction / 可能的修复方向

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Uh oh!

[Bug / Performance] 长音频 preset_spk_num 绕过 large-N 防护并触发极高开销的谱聚类 / Long audio with preset_spk_num bypasses large-N clustering safeguard #3514

Description

🐛 Bug

To Reproduce

Relevant FunASR code path

Code sample

Expected behavior

Error logs

Environment

Audio details

Additional investigation / 补充分析

Possible fix direction / 可能的修复方向

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions