temp_outputs/pred_rttms/mono_file.rttm File not found. Randomly happens, have to restart container.
**Describe the bug**

The bug is at a random interval we will suddenly get file not found or is not directory temp_outputs/pred_rttms/mono_file.rttm
Will work just fine hundreds of times and will randomly throw this.

**Steps/Code to reproduce bug**

Deploy as normal, using whisper + diarization (https://github.com/MahmoudAshraf97/whisper-diarization). Nothing special and it will work many times and just randomly will have that file missing. 

**Expected behavior**

Should not have a file missing...

**Environment overview (please complete the following information)**

Docker using nvcr.io/nvidia/pytorch:23.04-py3
If you have another container that nemo would work in then I am all ears.


**Additional context**

GPU: A100 
CUDA 11.7 with driver 525.XX
