Pegasus finetuning, should we always start with pegasus-large?
I’m fine-tuning pegasus on my own data, which is about 15,000 examples.I am finding, when fine-tuning Pegasus, usingpegasus-large, that the RAM requirements for even just abatch sizeof1are so extreme, that a Nvidia card with16GBof memory is required… just to run thebatch sizeof1! So at this point I am thinking that maybe my training will run better on the CPU, using a machine with a huge amount of ram… like512GBof ram… as this seems to allow a much bigger batch size, like up to64or128.My guess is that the RAM requirements are so extreme because I am usingpegasus-large. I’m doing this based on my understanding of this page::PegasusAll thecheckpointsare fine-tuned for summarization, besides pegasus-large, whence the other checkpoints are fine-tunedMy understanding from this is that, if we, as the newbie user, have some data we want to use with Pegasus, we should do this:Start with pegasus-large:google/pegasus-large · Hugging FaceFine tune it on our own dataUse thepytorch_model.binoutput from this fine tuning process to run inference on our own data.Am I getting something wrong here?  Given that I have 15,000 examples, have I made the correct determination that I should fine-tunepegasus-large, and that this will lead to the best results, even though the memory requirements are huge?I looked fordistilledmodel, here:Models - Hugging Face… But my understanding (possibly wrong?) is that thesedistilledmodels are ALREADY fine-tuned, so they would not be appropriate to use, given that I have a lot of my OWN data to fine-tune with.Thanks!
2020-12-22T06:26:51Z
To answer your second questions,thestudentmodels are a smaller version of the finetuned-pegasus models, created by choosing alternating layers from the decoder. This method is described in thePre-trained Summarization Distillation. To use these models you should fine-tune them
2020-12-22T13:16:45Z
valhalla:To use these models you should fine-tune themThanks for this. Shows that I am a newbie but I am still confused about this.Given that I have 15,000 examples, of my own data, to work with, I am still unclear if I should be usingpegasus-large, even though it has huge memory requirements, or if there is adistilledversion ofpegasusthat I can use for fine-tuning on my own data.It seems somehow illogical for me to START my fine tuning on a model that has ALREADY been fine-tuned on another task… especially because my summarization task doesn’t closely match the exists tasks likexsum,cnn, or etc.I have been studying ML now for 2 months, and this basic question has escaped me:  "Should I always start with alargemodel, and fine-tune that, or is it general practice to start with adistilledmodel, which has already been fine-tuned, and essentially make ANOTHER layer of fine-tuning, on top of that?
2020-12-22T13:19:28Z
And, yes, regarding thesestudentmodels, I have looked at them, like this one:huggingface.cosshleifer/student_pegasus_xsum_16_4 · Hugging FaceWe’re on a journey to advance and democratize artificial intelligence through open source and open science.But again, it looks like thesestudentmodels have already been fine-tuned, for example here, seems to be fine-tuned onxsum.
2020-12-23T05:58:43Z
it looks like thesestudentmodels have already been fine-tuned,These are created using the large fine-tuned model.You could also fine-tune the large model, and then distill that model into a student and fine-tune some more to get smaller model.
2024-05-03T10:48:37Z
I am at the same stage that you were in almost 4 years ago and I face the same doubt. I am not sure if I should begin with a larger model for fine-tuning against my data or use and already fine-tuned data. For more clear context I am working on data related to legal text summarization and thought Pegasus to be the appropriate model. Do you think I should opt for a different model?