Does task specific prefix matters for T5 fine-tuning?
If I understand correctly pre-trained T5 models were pre-trained with an unsupervised objective without any task specific prefix like “translate”, “summarize”, etc. Is it important then to create my summarization dataset for fine-tuning in a way that every input starts with "summarize: "?
2020-07-28T19:27:59Z
I think it is important, but am not totally certain why. You could test it pretty easily I bet.
2020-07-29T02:20:44Z
Actually, it may not be very important.I am confused about the prefix too and ask on the Github. And one of the code author replied. (there are more questions, you may benefit from it too as a beginner)https://github.com/huggingface/transformers/issues/6007
2020-07-29T05:44:14Z
I think the question is if the released pre-trained models were trained on a mixture of supervised and unsupervised tasks (so they learned with prefixes and putting them in front of my input when fine-tuning is useful) or only on unsupervised tasks. I know they do it both ways in the paper but not clear which one was released.
2020-07-29T06:07:12Z
Use prefix if your task is similar to one of the pre-trained tasks, otherwise it really doesn’t matter (at least I have observed this in my own experiments)
2020-07-29T16:30:51Z
Released models were pret-rained on both supervised and unsupervised tasks so prefix should help. Seehttps://github.com/google-research/text-to-text-transfer-transformer/issues/314#issuecomment-665764712
2020-07-29T16:32:26Z
Yes, that’s what I said, if task is similar to one of the tasks used in the pre-training mixture then it helps, if it’s completely different then it won’t matter
2020-07-29T16:35:37Z
Yes, that part was clear. I just wanted to clarify how the released models were trained. Although you are right, the answer was implicit in what you already said.
2020-07-29T16:40:59Z
Great part is that T5 performs really well with and without prefixHere’s what I observed in my experiments.,It converges slightly faster when using a task prefix and when the task was similar, say summarizationPerformed equally well even without prefix, took slightly longer to converge
2021-06-28T07:10:37Z
My question is related, how can I understand how prefix and prefix masking is working? where can I read the code related to processing the prefix? or there is nothing special about that? I mean it is simply adding a task in the config file? is the idea as simple as that?