Training interruption
Learning starts and then stops at a certain point.

![Screenshot_6](https://user-images.githubusercontent.com/82889232/199955482-b740d545-31a3-425b-ae63-709661b01e5f.png)

Termination of training without explanation, without logs.

![Screenshot_4](https://user-images.githubusercontent.com/82889232/199954667-d4cff613-d231-43c1-afe6-ac2ebeb7b6ef.png)

 Termination of training without explanation, without logs.

![Screenshot_5](https://user-images.githubusercontent.com/82889232/199954678-44718f78-4851-4a8d-8613-b22ecc47c08c.png)

> import nemo
import nemo.collections.asr as nemo_asr
import pytorch_lightning as pl
from nemo.utils.exp_manager import exp_manager
from omegaconf import DictConfig, OmegaConf

> params = OmegaConf.load(model_conf_path)

> min_duration = 0.5
epochs = 15
gpus = 1
lr_optim = 0.001

> params.trainer.devices = gpus
params.trainer.max_epochs = epochs
params.model.optim.lr = lr_optim
params.model.train_ds.manifest_filepath = train_manifest_path
params.model.train_ds.min_duration = min_duration
params.model.validation_ds.manifest_filepath = validation_manifest_path
params.model.test_ds.manifest_filepath = test_manifest_path
params.exp_manager.exp_dir = "/path/to/save_tr_nemo/"

> trainer = pl.Trainer(**params.trainer)
model = nemo_asr.models.EncDecCTCModel(cfg=params.model, trainer=trainer)
exp_manager(trainer=trainer, cfg=params.exp_manager)

> trainer.fit(model)
trainer.test(model)

There is yaml config:
https://github.com/Project-fil/Project-fil/blob/ee79825609cca9e8b01a9c4eb43b564bb5c2b35d/quartznet_ru.yaml

Cuda version: 11.6
Python version: 3.9.12
Torch version: 1.13.0a0+git35563f4
Nemo version: 1.11.0rc0
OS - Ubuntu 20.04.5 LTS
Graphic card - Nvidia A40

What could be wrong?



https://github.com/NVIDIA/NeMo/issues/5324#tasklist-block-cccbe04f-c2c1-4b01-a878-bd03a164bbc4


