Question: is it possible to reset the state of the streaming sortformer for a single speaker?

[MathieuDuponchelle]

Hi, thank you for releasing the streaming sortformer model, I am very enthusiastic about its performance.

One of my use cases is transcription of long, council meeting type audio. In such a situation, there will be very few (potentially only one) reoccuring speakers, and a large number of one-time speakers.

I wonder if flushing the state of the model for a single speaker (the "oldest" one) is feasible? If so, how would that be achieved?

Thanks again!

[MathieuDuponchelle]

Sorry for the "bug" label, I wasn't sure what category to choose, it can be removed by someone with sufficient permissions :)

[pzelasko]

CC @tango4j

[MathieuDuponchelle]

any input here? :)

[tango4j]

@MathieuDuponchelle
Sorry that I haven't respond in a timely manner.
And I appreciate a lot on showing your awareness on undeserved "bug" label, since often times these are feature requests.

The feature you are having in mind sounds feasible, but there are some caveats when you implement it.

You first need to find the dominant speaker's speaker index in that diarization session. If it is lucky, that speaker could be speaker index 0.

NeMo/nemo/collections/asr/modules/sortformer_modules.py

Lines 504 to 505 in 10e1597

	streaming_state.spkcache_preds = updated_spkcache_preds[:, : self.spkcache_len, :]

Then you acces spkcache_preds to remove any other speakers that is not your interest.
This can be understood by looking at picture that I attached.

If you are trying to remove the oldest one, in this picture, grab index 2 (python index),
Find the spkcache locations using the condition spkcache_preds == 2
Then finally remove those spkcache.

NeMo/nemo/collections/asr/modules/sortformer_modules.py

Lines 503 to 504 in 10e1597

	streaming_state.spkcache = updated_spkcache[:, : self.spkcache_len, :]
	streaming_state.spkcache_preds = updated_spkcache_preds[:, : self.spkcache_len, :]

I think you can create a rule-based code lines to do this operations; but since I haven't tried this so cannot guarantee that this could be successful.
But I suggest you to handle the variables that I pointed on.

[MathieuDuponchelle]

thanks @tango4j , no problem re timeliness you don't owe me anything :) I will experiment and report back once I have some time then, thanks again :)