The learning process is interrupted for no apparent reason
The training script seems to be correct. We don't do any network op during validation, but maybe if you're using wandb somehow that crashed and caused a disconnect ? I didn't install and use wandb. > The training script seems to be correct. We don't do any network op during validation, but maybe if you're using wandb somehow that crashed and caused a disconnect ?

Could this be an incompatibility of the NVIDIA H100 card with CUDA 11.8 version?
Do you have any other guesses please? This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
**Describe the bug**

I start training with a ready ckpt 'QuartzNet15x5_ru--val_wer=0.3912-epoch=116.ckpt'
I start the training with the command 'nohup python command file_name.py > info_about_train.txt 2>&1 &'
After some time (from 1 to 10 epochs), the training is simply interrupted, regardless of the stage (train or validation).

In 4 cases, the following error was recorded in the info_about_train.txt file:
`RuntimeError: DataLoader worker (pid some_pid) is killed by signal: Segmentation fault.`

In 5 cases, was for no stated reason:
`^MEpoch 5:   7%|▋         | 13081/175623 [57:20<11:52:33,  3.80it/s, v_num=4]`
`^MEpoch 6:  70%|██████▉   | 122281/175623 [9:05:50<3:58:`
`^MEpoch 9:  65%|██████▌   | 114210/175623 [9:13:05<4:57:24,  3.44it/s, v_num=10]`

In 2 cases:
`^MValidation DataLoader 0:  54%|█████▍    | 46838/86866 [2:45:06<2:21:06,  4.73it/s]^[[A^@^@^@^@^@^@^@^@^@^@`
`^MValidation DataLoader 0:   7%|▋         | 6260/86866 [07:05<1:31:23, 14.70it/s]^[[A^@^@^@^@^@^@^@^@^@^@^@`

In 1 cases:
`ConnectionResetError: [Errno 104] Connection reset by peer.`

**Please tell me what it could be?**

**Environment**
- Ubuntu: 20.04.6 LTS
- Conda: 23.3.1
- PyTorch: 2.1.2
- Python: 3.11
- NeMo: 1.21.0

GPU:
![Screenshot_1](https://github.com/NVIDIA/NeMo/assets/82889232/d8cacb02-0dc1-4dbf-96e9-da857cafd4a7)

Training start code: https://github.com/Project-fil/YamlForNeMo/blob/main/resume_from_ckpt.py