Nemotron-3-8b finetuning
Is `nemofw-training:23.08.03` subset of `nvcr.io/nvidia/nemo:23.08`? 

The official guide [here](https://docs.nvidia.com/nemo-framework/user-guide/latest/playbooks/llama2sft.html) suggests to use `nemofw-training:23.08.03`, but I got the following error when I `docker pull nvcr.io/ea-bignlp/ga-participants/nemofw-training:23.08.03`:
`Error response from daemon: pull access denied for nvcr.io/ea-bignlp/ga-participants/nemofw-training, repository does not exist or may require 'docker login': denied: requested access to the resource is denied`

Same error even after I docker logged in. Do you guys have any idea what is the reason and which one is supposed for SFT? Hi
Are you logging correctly?
You must register on the site below and use your created api-key to login.
ngc.nvidia.com @alireza0r Thank you for the reminder. Yes, it was related to the api-key. `docker login` command gave me login success info, but it didn't work. I have to use command `docker login nvcr.io -p api-key` to be able to successfully pull the image.  > @alireza0r Thank you for the reminder. Yes, it was related to the api-key. `docker login` command gave me login success info, but it didn't work. I have to use command `docker login nvcr.io -p api-key` to be able to successfully pull the image.

You actually need to have the proper access provided by NVIDIA. 
I was looking into this [newly released model](https://developer.nvidia.com/blog/nvidia-ai-foundation-models-build-custom-enterprise-chatbots-and-co-pilots-with-production-ready-llms/) and wanted to experiment with either SFT or PEFT-finetuning but it seems documentation is lacking on it (the [only tutorial](https://huggingface.co/nvidia/nemotron-3-8b-base-4k) available seems to be on inference at the moment). Actually, I couldn't find any mention to the model in the code so I'm not even sure training is possible at all with code available.

In support to above, I've tried the code using the latest available image (`nvcr.io/nvidia/nemo:23.08`) with a small-sized model (gpt-2) and it works fine. However, when plugging in the nemotron weights it raises an exception at [L1221](https://github.com/NVIDIA/TransformerEngine/blame/db589510a69b5b5c655d439ad3521ea890c7b454/transformer_engine/pytorch/attention.py#L1221) due to an unsupported Q,K,V layout. Note that this code no longer exist in main after [this PR](https://github.com/NVIDIA/TransformerEngine/pull/403).

Could you please confirm that the code available in this public repo _supports nemotron finetuning_ **and** provide some guidance on how to do it (e.g. pointer to the correct docker image, example usage would also be great)?

Tagging @ericharper and @nithinraok hoping they can help, thank you in advance!