Which version of transformer engine should I use, when I try to open ub_tp_comm_overlap?
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.
I am using NeMo with version v2.0.0rc0. When I set ub_tp_comm_overlap to true with tp and sp 2, I met the following error:
![Image](https://github.com/user-attachments/assets/8d78142c-2c64-4e8a-bdd8-71313485ef9c)

The version of transformer engine 1.6.0+c81733f. Should I update to newer te version?

When I update transformer engine to 1.13.0+e5edd6c. There occurs another error in NeMo:
![Image](https://github.com/user-attachments/assets/35d702a9-a599-4dab-99c6-42502e9f37c0)

CUDA 11.8 should be used in NeMo. But  transformer engine in 1.13.0+e5edd6c version requires CUDA newer than 12.0.

I'm stuck in these version issues.
Could you please tell me which version of NeMo and TE and CUDA should I use to enable tp_comm_overlap feature?