Fix incorrect embedding grads with distopt BF16 grad reductions
# What does this PR do ?

In the case where we are training GPT with distopt, BF16 grads, and pipeline parallelism, we are not properly synchronizing the embedding grads between the first and last pipeline stage. The embeddings were being accumulated into BF16 buffers and the pipeline-parallel reduction was on FP32 buffers (so just all-reducing zeros). Note that we synchronize these buffers before the data-parallel reduction, so that is still being done properly in FP32.

This PR fixes the issue by eagerly accumulating the embedding grads into FP32 buffers.

**Collection**: NLP

# Changelog 
- Distopt accumulates grads for FP32 params into FP32 buffers.

# Usage

In the config file, set the optimizer with `name: distributed_fused_adam` and `grad_sync_dtype: bf16`.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Depends/includes https://github.com/NVIDIA/NeMo/pull/6886
