srun issue with nemorun
Can you try adding `executor.srun_args = ["--mpi=pmix"]` and see if that works? @hemildesai 
How do I set that argument? I don't see it in the listed arguments with the command `nemorun llm pretrain --help`:

![Image](https://github.com/user-attachments/assets/d8af7cdf-4c57-4411-889e-859f52e5e581)
 In the current version of the repo, the `srun_args` ([here](https://github.com/NVIDIA/NeMo/blob/f27a982ed192f8039d5a30b51208847c9c1b0fc3/nemo/collections/common/parts/run_utils.py#L496)) seem to have the `--mpi=pmix` command by default? However, I still see the same error as before. This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
I am trying to run a simple pretraining job with nemorun: `nemorun llm pretrain --factory llama3_8b`

However, I see the following error before the training starts:
```
The application appears to have been direct launched using "srun",
but OMPI was not built with SLURM's PMI support and therefore cannot
execute. There are several options for building PMI support under
SLURM, depending upon the SLURM version you are using:

  version 16.05 or later: you can use SLURM's PMIx support. This
  requires that you configure and build SLURM --with-pmix.

  Versions earlier than 16.05: you must use either SLURM's PMI-1 or
  PMI-2 support. SLURM builds PMI-1 by default, or you can manually
  install PMI-2. You must then build Open MPI using --with-pmi pointing
  to the SLURM PMI library location.

Please configure as appropriate and try again.
```

I understand that this is a problem with the way openmpi is configured on my cluster but I can't change it.
An alternate solution that I have been told is to replace the `srun` command with something like `mpirun -np`. I want to try this but I am unable to locate where the `srun` command is being run in the first place? I dug into the [slurm.py file in NeMo-run](https://github.com/NVIDIA/NeMo-Run/blob/main/src/nemo_run/core/execution/slurm.py), but changing the commands there did not yield any difference.

Are there any other potential alternatives to this?