Error while multi-gpu training of megatron model
Hi, you can remove validation by either removing it from the YAML file or from CLI you can use the `~` operator.

```
~model.validation_ds
```

Could you give me some more information on your environment? Are you using an NVIDIA container?

Also could you clarify `Nemo branch - main (1.0.0rc1)`? main is a branch and so is r1.0.0rc1. Are you using the r1.0.0rc1 branch? How are you installing NeMo?

Thanks. Hi, @ericharper , Thanks for the reply, I'll try by removing validation,
I am using NVIDIA's PyTorch container version 20.12-py3 and installed nemo using `python -m pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[all]`. I mentioned 1.0.0rc1 because on doing `import nemo, nemo.__version__` I get this output. I also noticed now that there is an updated pytorch container, I'll try using that as well. Okay, if you wish to use the `main` branch, then please use the latest PyTorch container, 21.02.  Closing due to inactivity. Please reopen if still having the issue.
**bug description**

Hello,
This error report might seem little rough around the edges mainly because I've been trying out different language models in different nemo branches and within different environments too.
Starting with multi-gpu training of QAModel with _"megatron-bert-345m-uncased"_ lm on squadv2 dataset, gives me this error while its doing evaluation.

_Environment location - GCP instance (with 2 x Nvidia Tesla T4)
Nemo branch - main (1.0.0rc1)_
![image](https://user-images.githubusercontent.com/72965135/111428452-84fe6d00-871d-11eb-8ed9-b1c87ce9f675.png)

I am also getting this error when using DGX machine
_Environment location - NVIDIA DGX Station Version 4.7.0 (using 2 x Tesla V100)
Nemo branch - main (1.0.0rc1)_
![image](https://user-images.githubusercontent.com/72965135/111428864-1ec61a00-871e-11eb-9703-a98f1b8ba774.png)

Changing language model from megatron to "bert-base-uncased" does not give keyerror but gets stuck after running evaluation:
_Environment location - NVIDIA DGX Station Version 4.7.0 (using 2 x Tesla V100)
Nemo branch - main (1.0.0rc1)_
![image](https://user-images.githubusercontent.com/72965135/111434612-c430bc00-8725-11eb-9545-2c1a570c2ff9.png)

However I don't get any issue training on single gpu in this branch and also when training on multi-gpu in nemo's r1.0.0b3 branch, but I am not able to use megatron there. I found a similar bug report in huggingface transformers gitrepo ([#1472](https://github.com/huggingface/transformers/issues/1472)) where they mention workaround to not do evaluation while training, can this be done for QAModel too by its config file?
