Different outputs for a JIT operation
**Describe the bug**

I find that a JIT operation can have different outputs although the inputs are the same.

**Steps/Code to reproduce bug**

Suppose we have a checkpoint and load it to train a `MegatronGPTModel`. 

We print the inputs and outputs of this function: https://github.com/NVIDIA/NeMo/blob/v1.20.0/nemo/collections/nlp/modules/common/megatron/transformer.py#L556
```
layernorm_input = bias_dropout_add_func(attention_output, attention_bias, residual, self.hidden_dropout)
if torch.distributed.get_rank()==0:
    print(f"Layer: {self.layer_number} Attention checksum {layernorm_input.sum()}")
    print(f"Layer: {self.layer_number} Attention checksum {attention_output.sum()}")
    print(f"Layer: {self.layer_number} Attention checksum {attention_bias.sum()}")
    print(f"Layer: {self.layer_number} Attention checksum {residual.sum()}")
    print(f"Layer: {self.layer_number} Attention checksum {self.hidden_dropout}")
```
During the extra forward pass of the backward, we will get different outputs for this function despite the same inputs

For example, I get the following:
```
Layer: 1 Attention checksum -1.2981499433517456
Layer: 1 Attention checksum -0.10930676758289337
Layer: 1 Attention checksum 0.002560555934906006
Layer: 1 Attention checksum -1.1932671070098877
Layer: 1 Attention checksum 0
Layer: 2 Attention checksum -0.9067337512969971
Layer: 2 Attention checksum 0.4821169376373291
Layer: 2 Attention checksum 0.0022925958037376404
Layer: 2 Attention checksum -1.3912334442138672
Layer: 2 Attention checksum 0
Layer: 3 Attention checksum -1.014440894126892
Layer: 3 Attention checksum 0.02674156427383423
Layer: 3 Attention checksum -0.0017535686492919922
Layer: 3 Attention checksum -1.0347785949707031
Layer: 3 Attention checksum 0
Layer: 4 Attention checksum -1.6703314781188965
Layer: 4 Attention checksum -0.030592918395996094
Layer: 4 Attention checksum -0.0008977651596069336
Layer: 4 Attention checksum -1.6389871835708618
Layer: 4 Attention checksum 0
Layer: 1 Attention checksum -1.2990010976791382
Layer: 1 Attention checksum -0.10930676758289337
Layer: 1 Attention checksum 0.002560555934906006
Layer: 1 Attention checksum -1.1932671070098877
Layer: 1 Attention checksum 0
```
We can find that the outputs of `bias_dropout_add_func` in Layer 1 are different.

The key arguments I used are:
```
model.attention_dropout=0 
model.hidden_dropout=0 
model.transformer_engine=False
model.activations_checkpoint_granularity=full  
model.activations_checkpoint_num_layers=1
```

However, when setting `PYTORCH_JIT=0`, I will get the same outputs for `bias_dropout_add_func`.
**Expected behavior**

The function should always yield the same output.

**Environment overview (please complete the following information)**

I find this issue on NeMov1.20 docker

