Object shard /models/Nemotron-4-340B-Reward/model_weights/model.rm_head._extra_state/shard_0_1.pt not found
I am seeing the same issue, please help! 
cc @terrykong  This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
**Describe the bug**

Instruct model seems to work as expected, but I am experiencing issues with the Reward model.  https://huggingface.co/nvidia/Nemotron-4-340B-Reward/discussions/3

Tested containers: nemo_24_01_framework and nemo_24_03_01_framework
Resource config: #SBATCH --nodes=2 #SBATCH --ntasks-per-node=8 #SBATCH --gpus-per-node=8
Attempted resolution: Repeated downloading of the Nemotron-4-340B-Reward model
Related error: https://github.com/NVIDIA/NeMo/issues/8785

A clear and concise description of what the bug is.

NeMo reports that files are missing from the Nemotron-4-340B-Reward model.

**Steps/Code to reproduce bug**

Run the example code (https://huggingface.co/nvidia/Nemotron-4-340B-Reward) with one of the following containers: nemo_24_01_framework and nemo_24_03_01_framework


**Expected behavior**

A clear and concise description of what you expected to happen.

I expect the job to complete and not produce an error.

**Environment overview (please complete the following information)**

 Local H100 DGX cluster, using NGC containers,

