Slurm interactive mode, transcribe_speech_parallel.py gets stuck on consecutive runs
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
With container `nvcr.io/nvidia/nemo:24.07` (Pyxis/Enroot), ran in Slurm  interactive mode with 1 GPU if I execute the command
```bash
python3 /opt/NeMo/examples/asr/transcribe_speech_parallel.py \
...
```
the script will get stuck on the second of multiple consecutive runs. The point where it does so is 
```bash
> HERE
Using bfloat16 Automatic Mixed Precision (AMP)
GPU available: True (cuda), used: True
...
```

However if I execute the command
```bash
torchrun --standalone --nnodes=1 --nproc-per-node=1 /opt/NeMo/examples/asr/transcribe_speech_parallel.py \
...
``` 
I can run multiple consecutive times without any issues.

I have tried with standard debug parameters like 
```
TORCH_CPP_LOG_LEVEL=INFO 
TORCH_DISTRIBUTED_DEBUG=INFO 
NCCL_DEBUG=INFO 
NCCL_DEBUG_SUBSYS=ALL
```
but nothing peculiar pops out.