Precision Problem between nemo model and hugging face model
Hi, we are aware that some TE implementations won't generate identical results to those of HF (which uses native PyTorch). We use our fused version of operations to speed up training. It seems you are using the Llama model as a foundation model. NeMo thoroughly tests Llama models to ensure that even though the results are not bit-wise matching, the overall performance (benchmarks) is on par.

If you have more concerns about the behavior, please provide us with more details. What model are you converting, what command are you using, and how large is the gap? We can check whether the gap is reasonable. > Hi, we are aware that some TE implementations won't generate identical results to those of HF (which uses native PyTorch). We use our fused version of operations to speed up training. It seems you are using the Llama model as a foundation model. NeMo thoroughly tests Llama models to ensure that even though the results are not bit-wise matching, the overall performance (benchmarks) is on par.
> 
> If you have more concerns about the behavior, please provide us with more details. What model are you converting, what command are you using, and how large is the gap? We can check whether the gap is reasonable.

Thanks for your patient reply. We also test that the precission gab doesn't affect perfoemance.
**Describe the bug**

We are using nemo to training our large vision language model. When converting models from nemo format to hugging face format, we found that given the same inputs and weights, we get different outputs.

We test that after layer normalization, even using the same weights and input, outputs are different. 
Nemo use transformers engine and below code to calculate:
 
![飞书20240508-190258](https://github.com/NVIDIA/NeMo/assets/128022807/de2f959b-b6ce-4e13-ba81-272cbd5baaab)

And hugging face using pytorch
![image](https://github.com/NVIDIA/NeMo/assets/128022807/534d4b73-f991-47c2-9654-a6ed3d07f3f9)
 
And I also found there also exists little precision gap in rotational positional embedding、attention and FFN. 


**Expected behavior**

Is the precision gap caussed by different calculation operator? How can I fix that?

Thank you!
