Long-form audio speaker diarization OOM in clustering
Thanks for raising this issue, could you please attach full log here.  Attached the log from running NeuralDiarizer. 

[nemo.log](https://github.com/NVIDIA/NeMo/files/13428499/nemo.log)
 I'm also having this issue, hangs at this step, similar settings as op. Versions: Nemo 1.21.0, python 3.10

```
[NeMo I 2023-12-05 18:46:27 collections:302] Dataset loaded with 43 items, total duration of  0.60 hours.
[NeMo I 2023-12-05 18:46:27 collections:304] # 43 files loaded accounting to # 1 labels
vad: 100%|██████████| 43/43 [00:07<00:00,  6.06it/s]
[NeMo I 2023-12-05 18:46:34 clustering_diarizer:250] Generating predictions with overlapping input segments
``` @remenberl 
Thank you for uploading the samples. I will test it and get back to you.
Looking into the log you shared, I suppose 64GB RAM is not enough to handle 4 hours of diarization in an offline manner.
I will confirm the RAM requirement for this sample after run this myself. This worked for me after building from source, and setting num_workers to 0  This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
Hi,

Thanks for the recent development of long-form audio speaker diarization in https://github.com/NVIDIA/NeMo/pull/7737. Recently I encounter a 4-hour-long audio and observe OOM on RAM (not VRAM).

**Steps/Code to reproduce bug**
It can be reproduced by using audio https://podwise-hh.s3.us-west-1.amazonaws.com/0ef8d4f5beb504fae9f12272a83db030ab29c92f4e033eb86bfefe3d1668c7cf.m4a

My telephonic config file is just the default, with the clustering/msdd part pasted below:

```
  clustering:
    parameters:
      oracle_num_speakers: False
      max_num_speakers: 8
      enhanced_count_thres: 80
      max_rp_threshold: 0.25
      sparse_search_volume: 30
      maj_vote_spk_count: False 
      chunk_cluster_count: 50
      embeddings_per_chunk: 10000
  msdd_model:
    model_path: diar_msdd_telephonic
    parameters:
      use_speaker_model_from_ckpt: True 
      infer_batch_size: 25
      sigmoid_threshold: [0.7] 
      seq_eval_mode: False
      split_infer: True
      diar_window_length: 50
      overlap_infer_spk_limit: 5
```


**Expected behavior**
The job stops after my screen prints the last iteration of "Extracting embeddings for Diarization" and the program quickly consumes close to 64GB memory from 20GB- in previous steps. FYI, here are the last lines of prints before being killed.
```
[NeMo I 2023-11-19 20:54:29 clustering_diarizer:343] Extracting embeddings for Diarization
[NeMo I 2023-11-19 20:54:29 collections:445] Filtered duration for loading collection is  0.00 hours.
[NeMo I 2023-11-19 20:54:29 collections:446] Dataset loaded with 52949 items, total duration of  7.25 hours.
[NeMo I 2023-11-19 20:54:29 collections:448] # 52949 files loaded accounting to # 1 labels
```


**Environment overview (please complete the following information)**

 - Environment location: Ubuntu 22.04 with 64GB RAM, 4090 GPU
 - Method of NeMo install: `python -m pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[asr]`
 - If method of install is [Docker], provide `docker pull` & `docker run` commands used

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version: Ubuntu 22.04
- PyTorch version: '2.0.1+cu117'
- Python version: 3.10

**Additional context**

Add any other context about the problem here.
Example: GPU model
