Bugfix for BF16 grad reductions with distopt
# What does this PR do ?

https://github.com/NVIDIA/NeMo/pull/5920 adds support for BF16 grad reductions with distopt, with embedding grad reductions done in FP32. @mikolajblaz found some bugs and it turns out the FP32 reductions were not being done at all. This PR fixes those issues. When I run GPT-3 175B, I confirm the embedding grads are now optimized with the FP32 optimizer. Loss values are the same with FP32 and BF16 grad reductions, up to 50 steps and within numerical accuracy.

This turned out messier than I would have liked. It would have been better to integrate distopt support for multiple grad dtypes into Apex.

**Collection**: NLP

# Changelog 
- Debug distopt support for BF16 grad reductions

# Usage

Set the optimizer to `distributed_fused_adam` in the config file:

https://github.com/NVIDIA/NeMo/blob/aaa0ccacf5b4d204ed18f6052e55ff6f20575996/examples/nlp/language_modeling/conf/megatron_gpt_config.yaml#L207

Configure the optimizer with `grad_sync_dtype: bf16`.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* This PR is identical to https://github.com/NVIDIA/NeMo/pull/6297, but merges into `main` instead of `r1.17`. See that PR for discussions.
* Fixes bugs from https://github.com/NVIDIA/NeMo/pull/5920
* Depends on https://github.com/NVIDIA/apex/pull/1575
* Depends on https://github.com/NVIDIA/TransformerEngine/pull/75
