Evaluation without using a Trainer
Is there a clean way to perform evaluation on a model without using the Trainer?I have scripts which train and then evaluation my model. I want to perform further evaluation on the trained model - for now I’ve hacked my training script so I create a Trainer but don’t call train() - is there a better way of doing this?
2021-04-16T07:51:29Z
you’ll need to set up the evaluation dataset / dataloader and write your own evaluation loop to compute the metrics.you can find examples of this in theexamples/legacydirectory of thetransformersrepo, e.g. here is the evaluation for question-answering on squad:transformers/run_squad.py at 6e1ee47b361f9dc1b6da0104d77b38297042efae · huggingface/transformers · GitHub
2021-04-16T11:31:43Z
You can also have a look at our new examples that don’t use aTrainer(all therun_xxx_no_trainerin the task subfolders of the examples folder of the repo).