Pegasus Inference for production usecase
Hi,We have finetuneddistill-pegasus-cnn-16-4summarization model on our own data and results look good. However, when we want to deploy it for a real-time production use case - it is taking huge time on ml.c5.xlarge CPU (around 13seconds per document in a sequence). We tried ag4dn.xlargeGPU for inference and it is taking around 1.7seconds for one document in a sequence. Inference on a GPU is a costly affair. Could anyone suggest me if anyone have any ideas to make faster with less cost.I have tried “onnx runtime”,TorchScriptbut both of them dont have support for pegasus model. Do we have any timelines for supporting pegasus. I tried num_beams as well - above numbers are after using them.Inference code:import torchfrom transformers import PegasusForConditionalGeneration, PegasusTokenizersrc_text = [“”" PG&E stated it scheduled the blackouts in response to forecasts for high winds amid dry conditions. The aim is to reduce the risk of wildfires. Nearly 800 thousand customers were scheduled to be affected by the shutoffs which were expected to last through at least midday tomorrow."""]model_name = “sshleifer/distill-pegasus-cnn-16-4”torch_device = ‘cuda’ if torch.cuda.is_available() else ‘cpu’tokenizer = PegasusTokenizer.from_pretrained(model_name)model = PegasusForConditionalGeneration.from_pretrained(model_name).to(torch_device)inputs = tokenizer.batch_encode_plus(src_text, truncation=True, padding=“longest”, return_tensors=“pt”).to(torch_device)translated = model.generate(inputs[“input_ids”], num_beams=2)output = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in translated][0]print(output)Thanks in advance for your help. Apologies if I am missing something obvious.Regards,Karthik
2020-12-16T12:09:48Z
Hi,Could someone help in this regard?Thanks in advance,Karthik
2020-12-16T20:06:28Z
HI Karthik,Forgive me if you have looked into this, but have you tried model quantization?pytorch.orgQuantization — PyTorch 1.7.0 documentationIf ONNX is not supported, then your only other bet is to decrease the size of the model or change some parameters like you’ve been trying.
2020-12-17T03:28:28Z
Hi Karthik, one option would be to serve it with our hosted inference API for example with our startup plan here:https://huggingface.co/pricing. We would take care of the optimizations for you. If you have any question, feel free to write an email toapi-enterprise@huggingface.co.
2020-12-20T07:46:09Z
Thanks for your inputs. I will check them.
2020-12-20T14:42:16Z
@karthikgaliSomething is confusing about this post for me.  You wroteWe have finetuneddistill-pegasus-cnn-16-4But I can see in your code that you are loadingmodel_name = “sshleifer/distill-pegasus-cnn-16-4”If you fine-tuned it, wouldn’t you load your fine-tuned model, and not the model that has already been tuned bysshleifer?Put another way, see here:PegasusAll thecheckpointsare fine-tuned for summarization, besides pegasus-large, whence the other checkpoints are fine-tunedMy understanding from this is that, if we, as the newbie user, have some data we want to use with Pegasus, we should do this:Start with pegasus-large:google/pegasus-large · Hugging FaceFine tune it on our own dataUse thepytorch_model.binoutput from this fine tuning process to run inference on our own data.Note – I am not suggesting you are doing anything wrong. I’m just working with pegasus also, and reading your message makes me think that I’m maybe getting something wrong.Also – I am finding, when fine-tuning Pegasus, usingpegasus-large, that the RAM requirements for even just abatch sizeof1are so extreme, that a Nvidia card with16GBof memory is required… just to run thebatch sizeof1! So at this point I am thinking that maybe my training will run better on the CPU, using a machine with a huge amount of ram… like512GBof ram… as this seems to allow a much bigger batch size, like up to64or128.
2021-02-26T01:28:13Z
@karthikgaliDid you end up with a decent solution for this issue?  Did you try the solution suggested on StackOverflow below?Here are some leads:github.com/huggingface/transformersPegasus ONNX format?opened06:06AM - 06 Feb 21 UTCclosed03:03PM - 23 Apr 21 UTCAshwin367I tried to convert Pegasus to the ONNX format using [this ](https://colab.resear…ch.google.com/github/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb#scrollTo=foYlXrSksR_v) guide but it failed. Can Pegasus be converted to the ONNX format or is that not possible yet?stackoverflow.comhow to convert HuggingFace's Seq2seq models to onnx formatpython, tensorflow, pytorch, huggingface-transformersanswered bykiranron10:36AM - 09 Feb 21 UTCgithub.com/huggingface/transformersWhen would pegasus be able to be exported in ONNX format?opened10:37PM - 01 Nov 20 UTCclosed03:03PM - 24 Apr 21 UTCphosfuldevIt seems like it's not available now, I got this error:
`Error while converting…the model: Unrecognized configuration class <class 'transformers.configuration_pegasus.PegasusConfig'> for this kind of AutoModel: AutoModel.
Model type should be one of RetriBertConfig, T5Config, DistilBertConfig, AlbertConfig, CamembertConfig, XLMRobertaConfig, BartConfig, LongformerConfig, RobertaConfig, LayoutLMConfig, SqueezeBertConfig, BertConfig, OpenAIGPTConfig, GPT2Config, MobileBertConfig, TransfoXLConfig, XLNetConfig, FlaubertConfig, FSMTConfig, XLMConfig, CTRLConfig, ElectraConfig, ReformerConfig, FunnelConfig, LxmertConfig, BertGenerationConfig, DebertaConfig, DPRConfig, XLMProphetNetConfig, ProphetNetConfig.`

Which is fair since pegasus is a new addition. Is it something the team plans to do soon? 

Or can someone point me some resources on if there are other ways to export a pre-trained model from huggingface? I'm pretty new to the machine learning thing :p

Thanks all!thanks