EncDecSpeakerLabelModel to have full/close to full reproducibility
## Problem statement
I am training EncDecSpeakerLabelModel for speaker identification model , however I am **having difficulty on reproducibility** on model training.

Below code is part of the example from [HERE](https://github.com/NVIDIA/NeMo/blob/main/tutorials/speaker_tasks/Speaker_Identification_Verification.ipynb) with modification on seed.

## Main library 
Nemo version : v1.20.0
pytorch-lightning==1.9.4
torch==2.0.1


## Code
```python
import torch
import pytorch_lightning as pl
from pytorch_lightning.callbacks import ModelCheckpoint
from pytorch_lightning import seed_everything


accelerator = 'gpu' if torch.cuda.is_available() else 'cpu'
seed = 42

MODEL_CONFIG = os.path.join(NEMO_ROOT,'conf/titanet-large.yaml')
config = OmegaConf.load(MODEL_CONFIG)
checkpoint_callback = ModelCheckpoint(monitor='loss')
config.model.train_ds.manifest_filepath = train_manifest
config.model.validation_ds.manifest_filepath = validation_manifest
config.model.decoder.num_classes = 3
config.trainer.devices = 1
config.trainer.accelerator = accelerator
config.trainer.max_epochs = 5
config.trainer.strategy = None
config.model.train_ds.augmentor = None
config.trainer.deterministic = True

trainer_config = OmegaConf.create(dict(
    devices=1,
    accelerator=accelerator,
    max_epochs=5,
    max_steps=-1,  # computed at runtime if not set
    num_nodes=1,
    accumulate_grad_batches=1,
    enable_checkpointing=False,  # Provided by exp_manager
    logger=False,  # Provided by exp_manager
    log_every_n_steps=1,  # Interval of logging.
    val_check_interval=1.0,  # Set to 0.25 to check 4 times per epoch, or an int for number of iterations
    deterministic=True,
))
print(OmegaConf.to_yaml(trainer_config))

# Fix random seed in torch , numpy and dataloader.
torch.use_deterministic_algorithms(True)
seed_everything(seed, workers=True)

# Training config setting into pytorch lightning trainer
trainer = pl.Trainer(**config.trainer)


speaker_model = nemo_asr.models.EncDecSpeakerLabelModel(cfg=config.model, trainer=trainer)

trainer.fit(speaker_model)
trainer.validate(speaker_model)
```


## Output results 
**1st run :** 
![image](https://github.com/NVIDIA/NeMo/assets/13691221/5dc5daf8-8350-430b-bae0-531fa55b2c5c)

**2nd run:**
![image](https://github.com/NVIDIA/NeMo/assets/13691221/3ed7b250-6ab3-4f8e-ab39-03ade1f1d66e)

## Expected output 

Both runs should produce identical validation macro or close to identificatal validation macro scores.