[ASR] multi-node multi-gpu training hangs on end of epoch, post validation and checkpoint save
@KunalDhawan @nithinraok Have you seen anything like this? I had something similar, not sure if exactly the same though.

What I did was: 
1. Not use the container: Install everything manually inside a virtualenv.
2. Do not use `dali`. 
3. Check the callbacks you are using. 
 The cluster I'm running on does not allow bare metal installations. However, I'm also running the most recent NeMo container that is available on NGC (23.08), more precisely the example recipe  `examples/asr/asr_ctc/speech_to_text_ctc_bpe.py`. All of this under the assumption that apart from NCCL parameters due to local network specifics, it should work as is.  
 
In a 2 x 2 setting with 
```
export TORCH_CPP_LOG_LEVEL=INFO
export TORCH_DISTRIBUTED_DEBUG=DETAIL
```

I get the following error:
```py
...
RuntimeError: Detected mismatch between collectives on ranks. Rank 1 is running collective: CollectiveFingerPrint(SequenceNumber=76157OpType=BARRIER), but Rank 2 is running collective: CollectiveFingerPrint(SequenceNumber=0OpType=GATHER).Collectives differ in the following aspects:         Sequence number: 76157vs 0  Op type: BARRIERvs GATHER
...
RuntimeError: Detected mismatch between collectives on ranks. Rank 1 is running collective: CollectiveFingerPrint(SequenceNumber=76157OpType=BARRIER), but Rank 2 is running collective: CollectiveFingerPrint(SequenceNumber=0OpType=GATHER).Collectives differ in the following aspects:         Sequence number: 76157vs 0  Op type: BARRIERvs GATHER
...
RuntimeError: Detected mismatch between collectives on ranks. Rank 0 is running collective: CollectiveFingerPrint(SequenceNumber=76157OpType=BARRIER), but Rank 2 is running collective: CollectiveFingerPrint(SequenceNumber=0OpType=GATHER).Collectives differ in the following aspects:         Sequence number: 76157vs 0  Op type: BARRIERvs GATHER
...
RuntimeError: Detected mismatch between collectives on ranks. Rank 3 is running collective: CollectiveFingerPrint(SequenceNumber=76157, OpType=ALLREDUCE, TensorShape=[1], TensorDtypes=Float, TensorDeviceTypes=TensorOptions(dtype=float (default), device=cuda, layout=Strided (default), requires_grad=false (default), pinned_memory=false (default), memory_format=(nullopt))), but Rank 0 is running collective: CollectiveFingerPrint(SequenceNumber=0OpType=REDUCE).Collectives differ in the following aspects:    Sequence number: 76157vs 0  Op type: ALLREDUCEvs REDUCE
``` in your script, do you prepend your command with `srun`?

And I suppose you also added `SLURMEnvironment` to the `trainer`:

```py
from pytorch_lightning.plugins.environments import SLURMEnvironment
[...]
    trainer = pl.Trainer(
        **cfg.trainer,
        plugins=[SLURMEnvironment()],
    )
``` Yes, I do run with srun, and the SLURMEnvironment is properly set. 

I've discovered that the culprit is the addition of the parameter
```
+exp_manager.checkpoint_callback_params.save_best_model=true
```
as soon as I set `save_best_model` to `true` the above mentioned mismatch errors appear, I guess there must be something wrong with   
https://github.com/NVIDIA/NeMo/blob/937df1a4899d4a0ab9236ded491abd44a86e597c/nemo/utils/callbacks/nemo_model_checkpoint.py#L159-L184

Perhaps related to https://github.com/Lightning-AI/pytorch-lightning/issues/14302#issuecomment-1220525529, which has been corrected in `NeMoModelCheckpoint.on_tain_end()`, but doesn't seem to have been addressed here. Indeed I've noticed log warnings of the like
```
[NeMo W 2023-12-22 23:04:20 nemo_logging:349] /usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:433: PossibleUserWarning: It is recommended to use `self.log('global_step', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.

[NeMo W 2023-12-22 23:04:20 nemo_logging:349] /usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:433: PossibleUserWarning: It is recommended to use `self.log('val_loss', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.
 ```
 Could this be related? @itzsimpl is the above issue you described with `save_best_model` in exp_manager only in case of multi gpu runs ? Can it be reproduced with single GPU ? As said, single-node single-gpu, single-node multi-gpu works fine; the issue is present when running multi-node multi-gpu. I have the system busy at the moment, but for the sake of completeness I will re-check the case of multi-node single-gpu, asap, but if I recall correctly from previous experiments, it worked fine as well.

@titu1994 any ideas? This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. Turns out this issue pops out depending on the setup, the principal cause being the checkpoint reload that happens when `save_best_model` is used in combination with `always_save_nemo` (the error will/may happen on checkpoint):

https://github.com/NVIDIA/NeMo/blob/937df1a4899d4a0ab9236ded491abd44a86e597c/nemo/utils/callbacks/nemo_model_checkpoint.py#L169

and/or `save_last`(the error will/may happen on train end):

https://github.com/NVIDIA/NeMo/blob/937df1a4899d4a0ab9236ded491abd44a86e597c/nemo/utils/callbacks/nemo_model_checkpoint.py#L216

The checkpoint reload requires the files to exist on all nodes, as all nodes will reload it, even if only RANK 0 will actually save the `.nemo` archive (at least in the case of ASR and non-distributed checkpoints). 

When using a setup based on NFS shared storage, it is thus important to have the proper mounting parameters set on all nodes. NFS by default will cache directory entries and file attributes, which may prevent a file that was created on one node from being visible on another until the cache gets invalidated. In my testing the addition of `lookupcache=positive`, which sets the directory entry cache to always revalidate negative entries (non-existing files) was enough, but I haven’t tested with distributed-checkpoints. Others (like in #7756) suggest setting `lookupcache=none`, `noac` and/or `actimeo=0`, which disables directory entry and file attributes caching entirely, albeit with some performance penalty. 

 If changing the mounting parameters is not possible, one workaround is also to have `save_best_model` set to `false` (this resolves the issue at least for non-distributed checkpoints like in ASR training), but with the drawback that one needs to create the `.nemo` archive in a post-processing step. This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
I'm running a standard ASR training recipe (conformer ctc) in a multi-node multi-gpu setting using SLURM. The training stops on epoch end, post validation, when the checkpoint is saved. The next epoch never starts. Single-node multi-gpu training runs fine.

NCCL does not seem to be the problem (a simple dummy training based on MNIST runs fine, and the checkpoints are saved). I've also tried running the NeMo ASR code with `gloo` as backend, but there's no change, training hangs post validation after the model is saved. 

The shared storage, does not seem to be the issue, either, as single-node trainings work fine, so does training independently, but concurrently, on all nodes. Also, if I disable checkpointing training continues normally till the last epoch, when the end of training model is saved. 

The only clue I have is that I'm running using the `nemo:23.08` container, and I've noticed the following log
```python
[NeMo W 2023-11-25 21:45:41 nemo_logging:349] /usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/connectors/logger_connector/result.py:433: PossibleUserWarning: It is recommended to use `self.log('global_step', ..., sync_dist=True)` when logging on epoch level in distributed setting to accumulate the metric across devices.
```

Any ideas how to resolve this issue? 
