Validation not running when using `TarredTranslationDataset`
**Describe the bug**

The validation step is not running when training a [mt_enc_dec](https://github.com/NVIDIA/NeMo/blob/r1.2.0/nemo/collections/nlp/models/machine_translation/mt_enc_dec_model.py) model using `TarredTranslationDataset`. As a result, the checkpoint call back is not invoked and no validation metrics are reported. The initial validation sanity check runs fine and then training proceeds reporting a decrease in train_loss over time, but the validation is never invoked again.

**Steps/Code to reproduce bug**
I am using a cleaned wmt dataset with 3.6M rows and training using ddp.

Here are the relevant bits from the config:
```yaml
model:
  ...
  train_ds:
    src_file_name: ???
    tgt_file_name: ???
    use_tarred_dataset: true # if true tar_file_name and meta_file_name will be used (or created automatically)
    # config for preprocessing training data and creating a tarred datset automatically
    tar_file_prefix: parallel # prefix for tar file names
    tar_files: null # if data has already been preprocessed (rest of config ignored)
    metadata_file: null # metadata for tarred dataset
    lines_per_dataset_fragment: 1000000 # Number of lines to consider for bucketing and padding
    num_batches_per_tarfile: 100 # Number of batches (pickle files) within each tarfile
    tar_shuffle_n: 100 # How many samples to look ahead and load to be shuffled
    shard_strategy: scatter # tarred dataset shard distribution strategy
    n_preproc_jobs: -2 # number of processes to use for data preprocessing (-2 means all but 2)
    tokens_in_batch: 8000
    clean: true
    max_seq_length: 512
    shuffle: true
    num_samples: -1
    drop_last: false
    pin_memory: false
    num_workers: 0

  validation_ds:
    src_file_name: ???
    tgt_file_name: ???
    tokens_in_batch: 1024
    clean: false
    max_seq_length: 512
    shuffle: false
    num_samples: -1
    drop_last: false
    pin_memory: false
    num_workers: 8

trainer:
  gpus: 8
  num_nodes: 1
  max_steps: 500000
  amp_level: O2 # O1/O2 for mixed precision
  precision: 16 # Should be set to 16 for O1 and O2, default is 16 as PT ignores it when am_level is O0
  accelerator: ddp
  checkpoint_callback: false
  logger: false
  log_every_n_steps: 50  # Interval of logging.
  check_val_every_n_epoch: 1
```

Here is the screenshot of the logs:
![Screenshot 2021-09-13 at 16 51 47](https://user-images.githubusercontent.com/22484212/133116307-027ccbc0-978d-4437-bea7-d20414d2d4d6.png)

**Environment overview **

- Environment Location: Docker on Kubernetes
- Method of NeMo install: using helm chart using nvcr.io/nvidia/nemo:1.2.0

**Additional context**

Training on 8 A100 GPUs using DGX box

