AUTOTRAIN NOT working at all
I’ve tried to fine tune multiple models using many different datasets and once i click the start training button it turns red for a couple of seconds then turns blue again, I’ve tried this with multiple models and different datasets but nothing works, I’ve included the log file belowBlockquoteDevice 0: Tesla T4 - 434.4MiB/15360MiBINFO | 2024-06-27 09:25:29 | autotrain.app.utils:kill_process_by_pid:52 - Sent SIGTERM to process with PID 69INFO | 2024-06-27 09:25:29 | autotrain.app.utils:get_running_jobs:26 - Killing PID: 69ERROR | 2024-06-27 09:25:27 | autotrain.trainers.common:wrapper:121 - Error occurred while packing the dataset. Make sure that your dataset has enough samples to at least yield one packed sequence.ValueError: Error occurred while packing the dataset. Make sure that your dataset has enough samples to at least yield one packed sequence.raise ValueError(File “/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py”, line 627, in _prepare_packed_dataloaderreturn self._prepare_packed_dataloader(File “/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py”, line 519, in _prepare_datasettrain_dataset = self._prepare_dataset(File “/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py”, line 362, ininitreturn f(*args, **kwargs)File “/app/env/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py”, line 101, in inner_ftrainer = SFTTrainer(File “/app/env/lib/python3.10/site-packages/autotrain/trainers/clm/train_clm_sft.py”, line 44, in traintrain_sft(config)File “/app/env/lib/python3.10/site-packages/autotrain/trainers/clm/main.py”, line 28, in trainreturn func(*args, **kwargs)File “/app/env/lib/python3.10/site-packages/autotrain/trainers/common.py”, line 117, in wrapperTraceback (most recent call last):The above exception was the direct cause of the following exception:datasets.exceptions.DatasetGenerationError: An error occurred while generating the datasetraise DatasetGenerationError(“An error occurred while generating the dataset”) from eFile “/app/env/lib/python3.10/site-packages/datasets/builder.py”, line 1784, in _prepare_split_singlefor job_id, done, content in self._prepare_split_single(File “/app/env/lib/python3.10/site-packages/datasets/builder.py”, line 1627, in _prepare_splitself._prepare_split(split_generator, **prepare_split_kwargs)File “/app/env/lib/python3.10/site-packages/datasets/builder.py”, line 1122, in _download_and_preparesuper()._download_and_prepare(File “/app/env/lib/python3.10/site-packages/datasets/builder.py”, line 1789, in _download_and_prepareself._download_and_prepare(File “/app/env/lib/python3.10/site-packages/datasets/builder.py”, line 1027, in download_and_prepareself.builder.download_and_prepare(File “/app/env/lib/python3.10/site-packages/datasets/io/generator.py”, line 47, in read).read()File “/app/env/lib/python3.10/site-packages/datasets/arrow_dataset.py”, line 1125, in from_generatorpacked_dataset = Dataset.from_generator(File “/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py”, line 623, in _prepare_packed_dataloaderTraceback (most recent call last):The above exception was the direct cause of the following exception:KeyError: ‘text’self.formatting_func = lambda x: x[dataset_text_field]File “/app/env/lib/python3.10/site-packages/trl/trainer/utils.py”, line 480, inbuffer.append(self.formatting_func(next(iterator)))File “/app/env/lib/python3.10/site-packages/trl/trainer/utils.py”, line 503, initeryield from constant_length_iteratorFile “/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py”, line 620, in data_generatorfor idx, ex in enumerate(self.config.generator(**gen_kwargs)):File “/app/env/lib/python3.10/site-packages/datasets/packaged_modules/generator/generator.py”, line 30, in _generate_examplesfor key, record in generator:File “/app/env/lib/python3.10/site-packages/datasets/builder.py”, line 1748, in _prepare_split_singleERROR | 2024-06-27 09:25:27 | autotrain.trainers.common:wrapper:120 - train has failed due to an exception: Traceback (most recent call last):Generating train split: 0 examples [00:00, ? examples/s]Generating train split: 0 examples [00:00, ? examples/s]warnings.warn(/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:307: UserWarning: You passed adataset_text_fieldargument to the SFTTrainer, the value you passed will override the one in theSFTConfig.warnings.warn(/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:269: UserWarning: You passed amax_seq_lengthargument to the SFTTrainer, the value you passed will override the one in theSFTConfig.warnings.warn(/app/env/lib/python3.10/site-packages/transformers/training_args.py:1965: FutureWarning:--push_to_hub_tokenis deprecated and will be removed in version 5 ofTransformers. Use--hub_tokeninstead.warnings.warn(/app/env/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:181: UserWarning: You passed apackingargument to the SFTTrainer, the value you passed will override the one in theSFTConfig.warnings.warn(/app/env/lib/python3.10/site-packages/transformers/training_args.py:1965: FutureWarning:--push_to_hub_tokenis deprecated and will be removed in version 5 ofTransformers. Use--hub_tokeninstead.warnings.warn(message, FutureWarning)Deprecated positional argument(s) used in SFTTrainer, please use the SFTConfig to set these arguments instead./app/env/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:100: FutureWarning: Deprecated argument(s) used in ‘init’: dataset_text_field, max_seq_length, packing. Will not be supported from version ‘1.0.0’.INFO | 2024-06-27 09:25:27 | autotrain.trainers.clm.train_clm_sft:train:37 - creating trainerINFO | 2024-06-27 09:25:27 | autotrain.trainers.clm.utils:get_model:666 - model dtype: torch.float16Loading checkpoint shards: 100%|██████████| 2/2 [00:13<00:00, 6.98s/it]Loading checkpoint shards: 100%|██████████| 2/2 [00:13<00:00, 6.48s/it]Downloading shards: 100%|██████████| 2/2 [00:33<00:00, 16.96s/it]Downloading shards: 100%|██████████| 2/2 [00:33<00:00, 16.66s/it]low_cpu_mem_usagewas None, now set to True since model is quantized.. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.modeling_phi3.pyA new version of the following files was downloaded fromhttps://huggingface.co/microsoft/Phi-3-mini-4k-instruct:INFO | 2024-06-27 09:24:38 | autotrain.trainers.clm.utils:get_model:635 - loading model…INFO | 2024-06-27 09:24:38 | autotrain.trainers.clm.utils:get_model:627 - loading model config…WARNING | 2024-06-27 09:24:38 | autotrain.trainers.clm.utils:get_model:625 - Unsloth not available, continuing without it…INFO | 2024-06-27 09:24:38 | autotrain.trainers.clm.utils:get_model:583 - Can use unsloth: False. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.configuration_phi3.pyA new version of the following files was downloaded fromhuggingface.co/microsoft/Phi-3-mini-4k-instruct:warnings.warn(/app/env/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning:resume_downloadis deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, useforce_download=True.INFO | 2024-06-27 09:24:38 | autotrain.trainers.clm.utils:configure_block_size:548 - Using block size 1024INFO | 2024-06-27 09:24:38 | autotrain.trainers.clm.utils:configure_training_args:485 - configuring training argsINFO | 2024-06-27 09:24:38 | autotrain.trainers.clm.utils:configure_logging_steps:480 - Logging steps: 25INFO | 2024-06-27 09:24:38 | autotrain.trainers.clm.utils:configure_logging_steps:467 - configuring logging stepsSpecial tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.INFO | 2024-06-27 09:24:37 | autotrain.trainers.clm.utils:process_input_data:395 - Valid data: None})num_rows: 1110features: [‘question’, ‘answer’],INFO | 2024-06-27 09:24:37 | autotrain.trainers.clm.utils:process_input_data:394 - Train data: Dataset({Generating train split: 100%|██████████| 1110/1110 [00:00<00:00, 93188.10 examples/s]Generating train split: 0%| | 0/1110 [00:00<?, ? examples/s]Downloading data: 100%|██████████| 600k/600k [00:00<00:00, 20.6MB/s]Downloading data: 0%| | 0.00/600k [00:00<?, ?B/s]Downloading readme: 100%|██████████| 139/139 [00:00<00:00, 839kB/s]Downloading readme: 0%| | 0.00/139 [00:00<?, ?B/s]INFO | 2024-06-27 09:24:36 | autotrain.trainers.clm.train_clm_sft:train:12 - Starting SFT training…e[93m [WARNING] e[0m using untested triton version (2.3.0), only 1.0.0 is known to be compatiblee[93m [WARNING] e[0m sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3e[93m [WARNING] e[0m NVIDIA Inference is only supported on Ampere and newer architecturese[93m [WARNING] e[0m Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATHe[93m [WARNING] e[0m If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.e[93m [WARNING] e[0m async_io: please install the libaio-dev package with apte[93m [WARNING] e[0m async_io requires the dev libaio .so object and headers but these were not found.[2024-06-27 09:24:36,228] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)To avoid this warning pass in values for each of the problematic parameters or runaccelerate config.--dynamo_backendwas set to a value of'no'The following values were not passed toaccelerate launchand had defaults used instead:INFO | 2024-06-27 09:24:28 | autotrain.backends.local:create:13 - Training PID: 69INFO | 2024-06-27 09:24:28 | autotrain.commands:launch_command:401 - {‘model’: ‘microsoft/Phi-3-mini-4k-instruct’, ‘project_name’: ‘autotrain-8v782-tuct3’, ‘data_path’: ‘huggingfacepremium/train’, ‘train_split’: ‘train’, ‘valid_split’: None, ‘add_eos_token’: True, ‘block_size’: 1024, ‘model_max_length’: 2048, ‘padding’: ‘right’, ‘trainer’: ‘sft’, ‘use_flash_attention_2’: False, ‘log’: ‘tensorboard’, ‘disable_gradient_checkpointing’: False, ‘logging_steps’: -1, ‘eval_strategy’: ‘epoch’, ‘save_total_limit’: 1, ‘auto_find_batch_size’: False, ‘mixed_precision’: ‘fp16’, ‘lr’: 3e-05, ‘epochs’: 3, ‘batch_size’: 2, ‘warmup_ratio’: 0.1, ‘gradient_accumulation’: 4, ‘optimizer’: ‘adamw_torch’, ‘scheduler’: ‘linear’, ‘weight_decay’: 0.0, ‘max_grad_norm’: 1.0, ‘seed’: 42, ‘chat_template’: ‘none’, ‘quantization’: ‘int4’, ‘target_modules’: ‘all-linear’, ‘merge_adapter’: False, ‘peft’: True, ‘lora_r’: 16, ‘lora_alpha’: 32, ‘lora_dropout’: 0.05, ‘model_ref’: None, ‘dpo_beta’: 0.1, ‘max_prompt_length’: 128, ‘max_completion_length’: None, ‘prompt_text_column’: ‘prompt’, ‘text_column’: ‘text’, ‘rejected_text_column’: ‘rejected_text’, ‘push_to_hub’: True, ‘username’: ‘huggingfacepremium’, ‘token’: '*Blockquote****', ‘unsloth’: False}Blockquote