AssertionError: tensor and expert parallel group is not initialized
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
**Describe the bug**

Using `nemo-toolkit==1.23.0` and `megatron-core==0.5.0` will give the following error.
```
\  File \"/opt/conda/lib/python3.10/site-packages/megatron/core/parallel_state.py\"\
          , line 615, in get_tensor_and_expert_parallel_group\n    _TENSOR_AND_EXPERT_PARALLEL_GROUP\
          \ is not None\nAssertionError: tensor and expert parallel group is not initialized\n"
        timestamp: '1710051497'
      message: 'AssertionError: tensor and expert parallel group is not initialized'
```

A clear and concise description of what the bug is.

The reason is the following function calls:
[_set_random_seed(seed)](https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/modules/common/megatron/megatron_init.py#L167) -> [get_expert_model_parallel_rank()](https://github.com/NVIDIA/Megatron-LM/blob/89574689447d694bb19dd86fc8a6153b4467ba9d/megatron/core/tensor_parallel/random.py#L181C29-L181C59) -> [get_tensor_and_expert_parallel_group()](https://github.com/NVIDIA/Megatron-LM/blob/89574689447d694bb19dd86fc8a6153b4467ba9d/megatron/core/parallel_state.py#L946) -> [_TENSOR_AND_EXPERT_PARALLEL_GROUP](https://github.com/NVIDIA/Megatron-LM/blob/89574689447d694bb19dd86fc8a6153b4467ba9d/megatron/core/parallel_state.py#L617-L621).
For the last call, `_TENSOR_AND_EXPERT_PARALLEL_GROUP` is always None, since [initialize_model_parallel](https://github.com/NVIDIA/Megatron-LM/blob/89574689447d694bb19dd86fc8a6153b4467ba9d/megatron/core/parallel_state.py#L97) is not called.

