How to train T5 with Tensorflow
There are many examples showing how to train T5 in PyTorch, e.g.:https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb, but none so far for Tensorflow. Many people have been asking ontransformers:https://github.com/huggingface/transformers/issues/3626.Does anybody have a good notebook showing how to train T5 in Tensorflow?Otherwise, I will try to translate@valhalla's great notebook:https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynbto Tensorflow.
2020-08-09T03:17:57Z
Thanks very much Patrick@patrickvonplaten!In Suraj Patil’s notebook, he employed PytorchTrainerto train T5.At first, I didn’t know that we can useTrainerwith Seq2Seq problems (according to“The Big Table of Tasks”https://huggingface.co/transformers/examples.htmlwhich stated thatTrainerdoes not yet support Translation / Summarization )I will try to useTFTrainerfor TF2 on Seq2Seq problems. If that doesn’t work, I think I will try to write custom loop in TF2.
2020-08-09T08:37:19Z
You can use trainer for seq2seq as well, you’ll just need to write a different data collator which will return the expected arguments to the model.Few things to note about the that notebook,I wrote it before v3.0.0, few things have changed after thatDatCollatoris not aclassanymore, so you won’t need to inherit fromDataCollatorwhen creatingT2TDataCollator. Alsocollate_batchshould be renamed to__call__.lm_lablesis now deprecated, uselabelsinstead.Let me know if you run into problems using the notebook.
2020-08-29T12:10:14Z
Any luck on those Tensorflow T5 notebook?
2020-09-01T10:29:30Z
This might be related:How to train TFT5ForConditionalGeneration model?Okey, I will start working on a T5 TF notebook showing how T5 can be fine-tuned on CNN / Daily Mail using the TF Trainer this week.
2020-09-05T12:49:33Z
hey@patrickvonplateni want to contribute a fully working TF T5 training/finetuning notebook, how do i do that?
2020-09-07T06:15:33Z
Hey@HarrisDePerceptron,That sounds awesome! Usually people create a google colab and add it under community notebooks here:github.comtransformers/notebooks at main · huggingface/transformers - Community notebooks:main/notebooks🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.Looking forward to your notebook
2020-10-02T14:57:38Z
Hey everyone. We have recently contributed ourcommunity notebookthat lets us train T5 using pure tensorflow 2. do checkout it out !!! For any issue you can log them to ouroffical repo.
2022-10-27T12:07:45Z
Hello everyone.I am struggling to find a working notebook to illustrate training T5 using keras.@HarrisDePerceptronyour notebook doesn’t seem to run at the moment.@patrickvonplatenCan you point me to something useful?This thread seemed to be leading in that direction, but the trail has gone cold.Thanks!