Using evaluate.evaluator on a PEFT model
For an exercise I am experimenting with PEFT on GPT-2. One task is to evaluate the performance. For this I chose evaluate.evaluator, which worked fine on plain GPT-2. However, when trying to evaluate the fine-tuned model like thistask_evaluator = evaluator('text-classification')
eval_results = task_evaluator.compute(
    model_or_pipeline=lora_model,
    tokenizer=tokenizer,
    data=ds_test,
    input_column=dataset_textfield_name,
    metric=evaluate.combine(['accuracy', 'f1']),
    label_mapping=label2id
)I got the following error:AttributeError                            Traceback (most recent call last)File ~/.local/lib/python3.10/site-packages/peft/peft_model.py:529, in PeftModel.getattr(self, name)528 try:→ 529     return super().getattr(name)  # defer to nn.Module’s logic530 except AttributeError:File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1614, in Module.getattr(self, name)1613         return modules[name]→ 1614 raise AttributeError(“‘{}’ object has no attribute ‘{}’”.format(1615     type(self).name, name))AttributeError: ‘PeftModelForSequenceClassification’ object has no attribute ‘task’During handling of the above exception, another exception occurred:AttributeError                            Traceback (most recent call last)File ~/.local/lib/python3.10/site-packages/peft/tuners/lora/model.py:273, in LoraModel.getattr(self, name)272 try:→ 273     return super().getattr(name)  # defer to nn.Module’s logic274 except AttributeError:File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1614, in Module.getattr(self, name)1613         return modules[name]→ 1614 raise AttributeError(“‘{}’ object has no attribute ‘{}’”.format(1615     type(self).name, name))AttributeError: ‘LoraModel’ object has no attribute ‘task’During handling of the above exception, another exception occurred:AttributeError                            Traceback (most recent call last)Cell In[19], line 1----> 1 eval_results = task_evaluator.compute(2     model_or_pipeline=lora_model,3     tokenizer=tokenizer,4     data=ds_test,5     input_column=dataset_textfield_name,6     metric=evaluate.combine([‘accuracy’, ‘f1’]),7     label_mapping=label2id8 )9 print(eval_results)File ~/.local/lib/python3.10/site-packages/evaluate/evaluator/text_classification.py:134, in TextClassificationEvaluator.compute(self, model_or_pipeline, data, subset, split, metric, tokenizer, feature_extractor, strategy, confidence_level, n_resamples, device, random_state, input_column, second_input_column, label_column, label_mapping)130 data = self.load_data(data=data, subset=subset, split=split)131 metric_inputs, pipe_inputs = self.prepare_data(132     data=data, input_column=input_column, second_input_column=second_input_column, label_column=label_column133 )→ 134 pipe = self.prepare_pipeline(135     model_or_pipeline=model_or_pipeline,136     tokenizer=tokenizer,137     feature_extractor=feature_extractor,138     device=device,139 )140 metric = self.prepare_metric(metric)142 # Compute predictionsFile ~/.local/lib/python3.10/site-packages/evaluate/evaluator/base.py:474, in Evaluator.prepare_pipeline(self, model_or_pipeline, tokenizer, feature_extractor, device)472     if tokenizer is not None and feature_extractor is not None:473         logger.warning(“Ignoring the value of the preprocessor argument (tokenizerorfeature_extractor).”)→ 474 if (pipe.task != self.task) and not (self.task == “translation” and pipe.task.startswith(“translation”)):475     raise ValueError(476         f"Incompatiblemodel_or_pipeline. Please specifymodel_or_pipelinecompatible with the{self.task}task."477     )478 return pipeFile ~/.local/lib/python3.10/site-packages/peft/peft_model.py:531, in PeftModel.getattr(self, name)529     return super().getattr(name)  # defer to nn.Module’s logic530 except AttributeError:→ 531     return getattr(self.base_model, name)File ~/.local/lib/python3.10/site-packages/peft/tuners/lora/model.py:275, in LoraModel.getattr(self, name)273     return super().getattr(name)  # defer to nn.Module’s logic274 except AttributeError:→ 275     return getattr(self.model, name)File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1614, in Module.getattr(self, name)1612     if name in modules:1613         return modules[name]→ 1614 raise AttributeError(“‘{}’ object has no attribute ‘{}’”.format(1615     type(self).name, name))AttributeError: ‘GPT2ForSequenceClassification’ object has no attribute ‘task’What should I do to use evaluate.evaluator for fine-tuned models?
2024-04-10T16:47:15Z
@countermodeCould you find the error? Have the same here…