Fix FP8 scales conversion issue from non-nemo checkpoint to nemo checkpoint (try 2) (failed try 1: #6911)
# What does this PR do ?

TLDR; 
Converting a non-nemo checkpoint to NeMo with Transformer Engine modules results in truncating of scales in _extra_state. This WAR temporarily fixes the issue by manually toggling the fp8 attribute in Transformer Engine modules.

More context:
1. When exporting TE module data for checkpointing (fp8 scale info, etc.), `fp8` attribute needs to be set (see here: https://github.com/NVIDIA/TransformerEngine/blob/main/transformer_engine/pytorch/module/base.py#L344) but this attribute is only set during the forward (and when using `fp8_autocast` context manager). If the `fp8` attribute isn't set, the relevant scale information isn't included in the pickled `_extra_state` tensor (and this is the "truncation of scales" issue referred in the TLDR, basically an empty tensor is pickled which makes it seem like the scale information or `_extra_state` tensor is truncated). (Note that I tried adding the `fp8_autocast` context manager around the `model.save_to()` call and it didn't work as expected)
2. Until, there's a cleaner way to enable the `fp8` atribute for the Transformer Engine modules, this WAR allows externally enabling it before the `model.save_to()` call in the checkpoint converter script.


**Collection**: NLP

# Changelog 
- Toggle the `fp8` attribute in a module if it exists (if it exists, its probably a TE module)

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
