way to make inference Zero Shot pipeline faster?
HiCan you guys give me tips how to make Zero Shot pipeline inference faster?My current approach right now is reducing the model size/parameter(trying to train “base model” instead of "large model)Is there another approach?
2020-10-06T13:39:50Z
There’s some discussion inthis topicthat you could check out.Here are a few things you can do:Try out one of the community-uploadeddistilled modelson the hub (thx@valhalla) . I’ve found them to get pretty similar performance on zero shot classification and some of them are much smaller and faster. I’d start withvalhalla/distilbart-mnli-12-3(models can be specified by passing e.g.pipeline("zero-shot-classification", model="valhalla/distilbart-mnli-12-3")when you construct a model.If you’re on GPU, make sure you’re passingdevice=0to the pipeline factory in to utilize cuda.If you’re on CPU, try running the pipeline with ONNX Runtime. You should get a boost.Here’s a project(thx again@valhalla)  that lets you use HF pipelines with ORT automatically.If you have a lot of candidate labels, try to get clever about passing just the most likely ones to the pipeline. Passing a large # of labels for each sentence is really going to slow you down since each sentence/label pair has to be passed to the model together. If you have 100 possible labels but you can use some kind of heuristic or simpler model to narrow it down, that will help a lot.Use mixed precision. This ispretty easyif using PyTorch 1.6.
2020-10-06T15:30:52Z
thanks for the advice@joeddavand also thanks to@valhallafor amazing project about distiled modeldistiled model seems interesting…will try to look for it,and it would be great if its support more model,such as xlm-robertaanywayi’m trying to train/reproduce yourxlm-modelbut usingbase-modelinstead oflargeone to improve inference speed.will try usethis stepmaybeusing xlm-r because it support more languagesany tips that i should aware of ?thanks again
2020-12-23T03:45:42Z
joeddav:mixed precisionI noticed using the zero-shot-classification pipeline that loading the model (i.e. this line:  classifier = pipeline(“zero-shot-classification”, device=0)) takes about 60 seconds, but that inference afterward is quite fast.  Is there a way to speed up the model/tokenizer loading process?  Thanks!
2020-12-23T06:05:44Z
the pipeline actually loads the model twice, once in theget_frameworkfunction and then again on line 3296github.comhuggingface/transformers/blob/master/src/transformers/pipelines.py#L3296logger.warning("Model might be a PyTorch model (ending with `.bin`) but PyTorch is not available. ""Trying to load the model with Tensorflow.")if model_class is None:raise ValueError(f"Pipeline using {framework} framework, but this framework is not supported by this pipeline.")model = model_class.from_pretrained(model, config=config, revision=revision, **model_kwargs)if task == "translation" and model.config.task_specific_params:for key in model.config.task_specific_params:if key.startswith("translation"):task = keywarnings.warn('"translation" task was used, instead of "translation_XX_to_YY", defaulting to "{}"'.format(task),UserWarning,)For now, if you want, you could just hardcode the framework asptand remove the call toget_frameworkto load the model once.
2020-12-23T17:17:42Z
Seems like we should add a utility function tofile_utils.pyto check whether a tf/pt model file exists at a path without having to download it so that we don’t have to do this. Thoughts from@sguggeror@lysandremaybe?
2020-12-23T17:48:27Z
I’m confused about what the question is. You can pass a local path toXyzModel.from_pretrainedand it won’t download anything.
2020-12-23T18:07:34Z
Awesome thanks! – I made the change below and load time dropped from 61 second to 32 seconds:classifier = pipeline(“zero-shot-classification”, device=0)  ---->  classifier = pipeline(“zero-shot-classification”, framework=“pt”, device=0)
2020-12-23T18:26:29Z
@sguggerthe issue is whether the model is local or not the pipeline loads it twice, which adds up significant time for big models likebart-large
2020-12-23T18:38:47Z
Yes, so this has nothing to do withfiles_utils.
2021-01-07T14:37:08Z
I should have been clearer. The problem is that theget_frameworkfunction in the pipelines implementation determines the framework with a try/catch, attempting to load the model in pytorch and if it fails, loading it with tensorflow. But then it just throws the model away, so the pipeline constructor has to load it again later.What I was trying to say is: would it make sense to have a utility infile_utils.pythat can tell you whether a file exists without having to download the whole thing? In this case, it would allow us to check whether a model file exists for a particular framework (e.g.pytorch_model.bin) without having to wait for the large file to download if it does. I imagine that could be useful in other places too, but I’m not the expert, so I thought I’d see what you thoughtIf it wouldn’t be useful elsewhere, there’s probably an easier workaround without leavingpipelines.py.(sorry for the belated response)
2021-01-07T15:36:42Z
Understood now! This function could certainly be useful, yes.
2024-05-31T23:20:29Z
@joeddav@valhallaThanks for the amazing work on the original zero-shot classification model/pipeline and the distilled versions. I am a bit late to the party but I want to know if there are new techniques that I can apply to speed up inference on a single GPU if I am using valhalla/distilbart-mnli-12-3 or a fine-tuned version of that. I read through this guide:GPU inferencebut to be honest I am not sure if these can be applied to the valhalla/distilbart-mnli-12-3 model given that some layers have been removed during distillation. I would really appreciate some pointers from you guys. Thank you!
2024-06-03T14:14:43Z
Techniques which are often used to speed up models include quantization. I’d recommend taking a look at the Optimum library which implements a lot of optimizations for models available in the Transformers library, tailored towards various hardware providers:🤗 Optimum. ONNX is a framework often used by companies in production to speed up inference.
2024-11-24T18:19:10Z
set torch_dtype=torch.float16 likely faster, referencepipelines doc, pipe = pipeline(“text-generation”, model=model_id, torch_dtype=torch.float16, batch_size=2, device=0)