How do I evaluate a pretrained model on a test dataset?
Without training, how do I use a test dataset to evaluate a pretrained model?
2022-02-24T05:00:17Z
HelloAssuming you’re using PyTorch, you can wrap your model inside aTrainerand then calltrainer.evaluate(). An example (taken fromhere):from transformers import TrainingArguments

training_args = TrainingArguments("test_trainer"),


import numpy as np
from datasets import load_metric

metric = load_metric("accuracy")


def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)


trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=small_train_dataset,
    eval_dataset=small_eval_dataset,
    compute_metrics=compute_metrics,
)

trainer.evaluate()If you don’t want to use aTrainer, check out the exampleshereand check the files ending with_no_trainer.py.