Evaluating your model on more than one dataset
Hi,Transformer’s Trainer and Trainingarguments classes allow for only one dataset to use for evaluation. Is there a simple way of adding another one? So, after after an epoch of training my model I could evaluate it on both training and developmental datasets and get metrics for both of them as one output? I know I could alter the training_args.py or trainer.py but I am pretty sure I would only mess things up…
2020-10-15T13:29:32Z
I think the easiest way to do this is to use the new system ofTrainerCallbackand write a callback that performs a new evaluation on your other datasets during the eventon_validate.
2021-01-04T10:28:30Z
Is it possible to provide an example?
2022-02-28T16:58:25Z
@sgugger, I had a brief look at the interfaces provided to achieve this, but I don’t see how this is possible. As I understand, theTrainerCallbackclass receives theTrainerArgumentswhich are used to initialise theTrainerbut this class does not allow us to pass additionaleval_dataloaders. Therefore, I am not sure how to pass the additional datasets to theTrainerin the first place.