Tarred Datasets for NMT
This pull request **introduces 4 alerts** when merging 8804196b70e585ebb8bd2847f4a503ba19358ac2 into cd8f1c5a74e0d0f2c5bf5847951e30d1170909c3 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-33c1b211c8938e3824030f9c8b2b1d9708c3bfdb)

**new alerts:**

* 3 for Unused import
* 1 for Unused local variable This pull request **introduces 4 alerts** when merging e6d59380c7f599616d043750c4d58eefd001a7f2 into cd8f1c5a74e0d0f2c5bf5847951e30d1170909c3 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-b707f2a3fe60823dab01087fadc6822aac488bf3)

**new alerts:**

* 4 for Unused import This pull request **introduces 1 alert** when merging 8d0d83dd2b36e1d250f2aacbb63471b7dbc00127 into cd8f1c5a74e0d0f2c5bf5847951e30d1170909c3 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-eb9c72b8bb41b91e29b388e0b4010687909a0fb5)

**new alerts:**

* 1 for Unused import @MaximumEntropy Try the final dataloader on a multigpu setup to full convergence before merge. Otherwise, it looks good.  this should go to "main" (I should probably delete nmtmodel branch). And now there are a bunch of merge conflicts. Any thoughts why? @okuchaiev I guess its because this branch was created from nmtmodel rather than main and there have been some commits to main after the nmtmodel merge (like typing). I can either resolve these conflicts or just submit another PR starting from main. Closing this PR since it started from the wrong branch. New one at #1683 
This PR adds support for Tarred Datasets when training NMT models on very large datasets. It avoids loading the entire dataset into CPU memory for every DDP process. It uses webdataset (https://github.com/tmbdev/webdataset) similar to ASR and language modeling.

To use this functionality

1. Run `python examples/nlp/machine_translation/create_tarred_dataset.py --clean --bpe_dropout 0.1 --shared_tokenizer --src_fname src_file.txt --tgt_fname tgt_file.txt --out_dir /path/to/out_dir --vocab_size 32000 --tokens_in_batch 16000 --lines_per_dataset_fragment 5000000 --num_batches_per_tarfile 1000` to create a folder that contains many tar files. Each tar file consists of a certain number of batches (specified by `num_batches_per_tarfile`), where each batch is a pickle file. The output folder will also contain `metadata.json` that contains the total number of training batches and YTTM tokenizer model files.
2. Point to the tarred dataset folder when training like so

```
python examples/nlp/machine_translation/enc_dec_nmt.py -c aayn_base \
     trainer.gpus=8 \
     +trainer.max_steps=300000
     ........... 
     model.train_ds.src_file_name=/path/to/out_dir/batches.tokens.16000._OP_1..100_CL_.tar \
     model.train_ds.tgt_file_name=/path/to/out_dir/batches.tokens.16000._OP_1..100_CL_.tar \
     +model.train_ds.load_from_tarred_dataset=true \
     +model.train_ds.metadata_path=/path/to/out_dir/metadata.json \
     +model.train_ds.tar_shuffle_n=100 \
     .............
```