MBART50 .generate() is very slow
Hello,I am currently working on theMBART50 many-to-onemodel for translation. The model takes a really long time to generate the translation.Is this normal? How can we optimize it?I tried in CPU and GPU but both remain slow :model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-one-mmt")Inference time in seconds formodel.generate(**input, max_length=max_length)whereinputis a tokenized string with 1024 tokens :max_length8 CPUs1 GPU200~38s~4s512~105s~11s750~160s~16s1024~237s~22sIt takes this long just for one string… Doing it in batch does not make it faster.Any idea what’s wrong or how to optimize?Thank you !