LinAlg Error When Running Clustering Diarizer
**Describe the bug**

Diarization continues until clustering, upon which I get this error.
"linalg.eigh: The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated eigenvalues (error code: 529)"

**Steps/Code to reproduce bug**

I'm loosely following the notebook at https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Diarization_Inference.ipynb except i turned oracle vad off.

Here's the diarization part of my notebook
https://colab.research.google.com/drive/1zN9LZ5-hPEI15K1NWl2obvEapJbTWd9P?usp=sharing
