BART-MNLI performance optimization
Hi, everyone!I need to classify texts of 100-words length on average into 1.5k classes in zero-shot setting.To solve this task I am usingfacebook/bart-large-mnlimodel.My setup is 32 CPU, 250 RAM.On the first two pictures below you can see memory consumption during model inference. On both pics I categorize only 4 texts. As you can see time and memory consumption grow with text length. It seems to me, corresponding complexities are O(n) and O(log n), where n is the number of words in text.I wonder if such big resource consumption is expected?  On average to classify just one 90-word text you need to have more than 50Gb RAM and it takes more than 3 minutes. On 32 CPUs!I wish I could use my NVIDIA P106 with 6Gb memory to speed up the inference. But this process doesn’t fit into its memory. Also, you can see that I triedvalhalla/distilbart-mnli-12-1. Evidently, I can’t use it with my GPU either. Are there any workarounds? And what is the underlying mechanism of such consumption? It it attention?Also, I tried to convert these MNLI models into ONNX to speed up inference. In that I miserably failed. I  found those few tutorials I could understand and implemented the steps suggested. The problem is that  all resulted onnx-models don’t have all necessary inputs. It hasinput_idsandattention_maskwhich, to my knowledge, responsible for text I want to categorize, but missesdecoder_input_idsanddecoder_attention_maskwhich, to my knowledge, responsible for candidate labels. I triedtransformers.onnxandonnx_transformersby@valhallaunsuccessfully. Though latter, after resolving many package conflicts, ran with roberta-mnli. The problem is that it ran only on one CPU out of 32 and I didn’t find way to enable the others. Is there any chance somebody here could guide me how properly transform bart-mnli into onnx to make inference? Will it help if I pay for subscription to get such help?Thank you.image841×475 47.2 KBimage (1)843×472 43.4 KBimage (2)838×473 81.2 KB
2022-05-13T13:35:43Z
No ideas at all?
2022-06-28T17:32:40Z
Did you find a solution to your problem? I will be interested to see how you solve it.
2024-01-24T23:55:28Z
Also looking for solutions to increase the speed of summarization using the BART model, did you find a solution?