Trainer.evaluate() with text generation
Hi everyone, I’m fine-tuning XLNet for generation.  For training, I’ve edited the permutation_mask to predict the target sequence one word at a time.  I’m evaluating my trained model and am trying to decide between trainer.evaluate() and model.generate().  Running the same input/model with both methods yields different predicted tokens.  Is it correct that trainer.evaluate() is not set up for sequential generation?  I’ll switch my evaluation code to use model.generate() if that’s the case. Thanks for the help!
2020-08-11T14:25:47Z
Hi, I encountered a similar problem when trying to use EncoderDecoderModel for seq2seq tasks. It seems like Trainer does not support text-generation tasks for now, as their websitehttps://huggingface.co/transformers/examples.htmlshows.
2020-08-11T14:28:56Z
There’s aPRfor that, you can try to use it.
2021-12-31T09:42:36Z
Is there any update regarding this topic?I would like to train aVisionEncoderDecoderModelfor image captioning and measure the BLEU metrics during evaluation. TheEvalPredictionobject I get incompute_metricsjust contains the logits, not the generated texts or tokens (i.e. the result of a beam search). I would assume that the computation of metrics on the result ofgenerateis not uncommon.ThePRmentioned in this thread seems to be stale and there have been quite some changes toTrainersince it was proposed.
2021-12-31T14:44:17Z
Hi@cgawron, you can take a look at my TrOCR notebooks here:Transformers-Tutorials/TrOCR at master · NielsRogge/Transformers-Tutorials · GitHub.They include several example notebooks regarding fine-tuning TrOCR (which is an instance ofVisionEncoderDecoderModel). I have a notebook regarding using the Seq2SeqTrainer, but also using native PyTorch. In both cases, I illustrate how to compute metrics usinggenerate(in the notebooks I use CER, but you can easily replace it with something like ROUGE or BLEU).
2021-12-31T19:57:14Z
Thank you,@nielsr, for providing these examples!Just for reference for other readers:TheSeq2SeqTrainingArgumentsnow contain a flagpredict_with_generateexactly for this purpose.