Add EMA support to NeMo
# What does this PR do ?

Adds Exponential Moving Average (EMA) support. We've seen promising results in reducing training times to reach convergence.

Things to do:

- [x] Capture speed degradation using the EMA callback.

There is practically none when the EMA weights are already on CUDA. Below are convergence results using LibriSpeech (SP) and the Conformer RNNT config for ASR:

**Precision 32**

WER vs Global Step

![Screenshot 2022-09-14 at 15 33 17](https://user-images.githubusercontent.com/6707363/190184437-2d5f043a-7e32-4a46-9f02-cea4ead8556c.png)

WER vs Process Time

![Screenshot 2022-09-14 at 15 34 40](https://user-images.githubusercontent.com/6707363/190184580-c71e9a90-7e0b-4fd4-8915-bf817180f2a3.png)

**Precision 16**

WER vs Global Step

![Screenshot 2022-09-14 at 15 33 43](https://user-images.githubusercontent.com/6707363/190184495-a2d1c692-0e14-4969-9763-6a5d443ca490.png)

WER vs Process Time

![Screenshot 2022-09-14 at 15 34 17](https://user-images.githubusercontent.com/6707363/190184541-911e7042-342a-4dfc-97b6-fe349aff14e9.png)

- [x] Explore keeping EMA weights on the CPU (transfer to GPU just for the multi-tensor apply, then bring it back to CPU)
- [x] Test using Quartznet/Conformer to ensure we see expected improvements using the EMA callback
- [ ] Add multi-node/gpu correctness test (EMA weights should be the same across processes) to jenkinsfile

**Collection**:
- Common

# Changelog 
- Adds Exponential Moving Average (EMA) support.

# Usage

The callback can be added to pure lightning, however it requires Apex to be installed (for the multi-tensor apply).

```python
...
lenet5 = MNISTLeNet5(MNISTLeNet5Config())
lenet5.setup_training_data()
lenet5.setup_optimization()

trainer = Trainer(
    accelerator='gpu',
    devices=1,
    callbacks=EMA(ema=0.999)
)
trainer.fit(model=lenet5)
```

Currently I've exposed this through the exp manager (questionable if this is the right approach or not) to use through the hydra configs.
```bash
python examples/asr/asr_ctc/speech_to_text_ctc.py \
    model.train_ds.manifest_filepath=... \
    model.validation_ds.manifest_filepath=... \
    trainer.devices=1 \ 
    +exp_manager.enable_ema=True
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

cc @titu1994 
