NeMo docker image 24.09 shows weird FileNotFoundError for shards FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpxoyazmhf/model_weights/model.decoder.layers.self_attention.core_attention._extra_state/shard_0_32.pt'
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale. Did you solve this? 

Hello everyone and Nvidia folks,


We are trying to finetune llama3_1_8b_instruct. (70b as well after 8b)  

Docker pull Nvidia NeMo docker image 24.09 and run inside a container.

Using this notebook from Brev (now part of Nvidia) https://github.com/brevdev/notebooks/blob/main/llama31_law.ipynb

The error below happened when running the 1st cell at step 2 in the notebook for finetuning 

![Image](https://github.com/user-attachments/assets/14a4b19b-82f7-4d0d-a6d6-09b1c412cb35)

We are running this on our local machine with 4* A10 GPUs. 

Tried both 1 GPU and 4 GPUs but got the same FileNotFoundError for shard

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpxoyazmhf/model_weights/model.decoder.layers.self_attention.core_attention._extra_state/shard_0_32.pt'


Anyone knows why this happened and how to solve? 
