Problem running LoRA PEFT on Llama 3 8B Instruct using NeMo docker container
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
I am trying to run the tutorial on [LoRA PEFT on Llama 3 8B Instruct using PubMedQA with NeMo](https://github.com/NVIDIA/NeMo/tree/main/tutorials/llm/llama-3/biomedical-qa) . I am running the notebook through the provided docker container, but I'm encountering some errors right at the early start of the code. Both the model and datasets seem to be properly loaded into the container.

When running:
%%bash

# Set paths to the model, train, validation and test sets.
MODEL="./workspace/llama-3-8b-instruct-nemo_v1.0/8b_instruct_nemo_bf16.nemo"
TRAIN_DS="[./pubmedqa/data/pubmedqa_train.jsonl]"
VALID_DS="[./pubmedqa/data/pubmedqa_val.jsonl]"
TEST_DS="[./pubmedqa/data/pubmedqa_test.jsonl]"
TEST_NAMES="[pubmedqa]"

SCHEME="lora"
TP_SIZE=1
PP_SIZE=1

OUTPUT_DIR="./results/Meta-Llama-3-8B-Instruct/"
rm -r $OUTPUT_DIR

torchrun --nproc_per_node=1 \
/opt/NeMo/examples/nlp/language_modeling/tuning/megatron_gpt_finetuning.py \
    exp_manager.exp_dir=${OUTPUT_DIR} \
    exp_manager.explicit_log_dir=${OUTPUT_DIR} \
    trainer.devices=1 \
    trainer.num_nodes=1 \
    trainer.precision=bf16-mixed \
    trainer.val_check_interval=20 \
    trainer.max_steps=500 \
    model.megatron_amp_O2=True \
    ++model.mcore_gpt=True \
    model.tensor_model_parallel_size=${TP_SIZE} \
    model.pipeline_model_parallel_size=${PP_SIZE} \
    model.micro_batch_size=1 \
    model.global_batch_size=8 \
    model.restore_from_path=${MODEL} \
    model.data.train_ds.num_workers=0 \
    model.data.validation_ds.num_workers=0 \
    model.data.train_ds.file_names=${TRAIN_DS} \
    model.data.train_ds.concat_sampling_probabilities=[1.0] \
    model.data.validation_ds.file_names=${VALID_DS} \
    model.peft.peft_scheme=${SCHEME}


I get the following error:
[NeMo W 2024-09-09 10:39:45 exp_manager:966] The checkpoint callback was told to monitor a validation value and trainer's max_steps was set to 500. Please ensure that max_steps will run for at least 1 epochs to ensure that checkpointing will not error out.
Error executing job with overrides: ['exp_manager.exp_dir=./results/Meta-Llama-3-8B-Instruct/', 'exp_manager.explicit_log_dir=./results/Meta-Llama-3-8B-Instruct/', 'trainer.devices=1', 'trainer.num_nodes=1', 'trainer.precision=bf16-mixed', 'trainer.val_check_interval=20', 'trainer.max_steps=500', 'model.megatron_amp_O2=True', '++model.mcore_gpt=True', 'model.tensor_model_parallel_size=1', 'model.pipeline_model_parallel_size=1', 'model.micro_batch_size=1', 'model.global_batch_size=8', 'model.restore_from_path=./workspace/llama-3-8b-instruct-nemo_v1.0/8b_instruct_nemo_bf16.nemo', 'model.data.train_ds.num_workers=0', 'model.data.validation_ds.num_workers=0', 'model.data.train_ds.file_names=[./pubmedqa/data/pubmedqa_train.jsonl]', 'model.data.train_ds.concat_sampling_probabilities=[1.0]', 'model.data.validation_ds.file_names=[./pubmedqa/data/pubmedqa_val.jsonl]', 'model.peft.peft_scheme=lora']
Traceback (most recent call last):
  File "/opt/NeMo/examples/nlp/language_modeling/tuning/megatron_gpt_finetuning.py", line 62, in main
    model_cfg = MegatronGPTSFTModel.merge_cfg_with(cfg.model.restore_from_path, cfg)
  File "/opt/NeMo/nemo/collections/nlp/parts/mixins/nlp_adapter_mixins.py", line 469, in merge_cfg_with
    base_cfg = cls.restore_from(path, return_config=True)
  File "/opt/NeMo/nemo/collections/nlp/models/nlp_model.py", line 465, in restore_from
    return super().restore_from(
  File "/opt/NeMo/nemo/core/classes/modelPT.py", line 464, in restore_from
    instance = cls._save_restore_connector.restore_from(
  File "/opt/NeMo/nemo/collections/nlp/parts/nlp_overrides.py", line 1122, in restore_from
    loaded_params = super().load_config_and_state_dict(
  File "/opt/NeMo/nemo/core/connectors/save_restore_connector.py", line 143, in load_config_and_state_dict
    self._unpack_nemo_file(
  File "/opt/NeMo/nemo/core/connectors/save_restore_connector.py", line 597, in _unpack_nemo_file
    members = [x for x in tar.getmembers() if ".yaml" in x.name]
  File "/usr/lib/python3.10/tarfile.py", line 1987, in getmembers
    self._load()        # all members, we first have to
  File "/usr/lib/python3.10/tarfile.py", line 2682, in _load
    tarinfo = self.next()
  File "/usr/lib/python3.10/tarfile.py", line 2589, in next
    raise ReadError("unexpected end of data")
tarfile.ReadError: unexpected end of data

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
[2024-09-09 10:39:48,947] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 74718) of binary: /usr/bin/python
Traceback (most recent call last):
  File "/usr/local/bin/torchrun", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 812, in main
    run(args)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 803, in run
    elastic_launch(
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 135, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 268, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/opt/NeMo/examples/nlp/language_modeling/tuning/megatron_gpt_finetuning.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2024-09-09_10:39:48
  host      : ip-172-32-0-242.eu-west-1.compute.internal
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 74718)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
