Gpt2 inference with onnx and quantize
Hey guys,I’ve managed to create a quantize version of gpt2 using onnxruntime but i don’t seem to be able to run it for some reason. anyone has a tutorial for it? also how does the “generate” method of the model will work with that any ideas?
2021-02-01T08:42:25Z
Hi@yanagar25when you say you cannot run the quantized version, what kind of error are you running into?Here’s a notebook that explains how to export a pretrained model to the ONNX format:transformers/04-onnx-export.ipynb at master · huggingface/transformers · GitHubYou can also find more details here:Exporting transformers models — transformers 4.2.0 documentationI don’t see an obvious reason why thegeneratemethod should not work after quantization, so as with most things in deep learning the best advice is to just try and see if it does
2021-02-02T11:15:41Z
Hey,@lewtunI am sorry if i am asking something that have been answered before. but in order to run the quantized model i need to run it withonnxruntime.InferenceSession. How does that can be combined with using thegeneratemethod? from what i understand i need to copy the entire logic from thegeneratemethod and instead of usingself(...)usesession.run(None, ort_inputs). please correct me if i’m wrong
2021-02-03T10:09:08Z
Ah now I understand better what you’re trying to achieve. Indeed you might have to write your owngeneratemethod so that you can integrate theInferenceSession- there’s an example of doing text generation with GPT-2 in the ONNX repo here:onnxruntime/Inference_GPT2_with_OnnxRuntime_on_CPU.ipynb at master · microsoft/onnxruntime · GitHubYou could just adapt their approach to include the generation method you need (beam search, sampling etc)
2021-02-03T10:43:24Z
thank you so much for the reply!
2021-02-03T10:58:02Z
FYI there’s a nice section in the docs that explains the various text generation strategies and how they’re implemented:Utilities for Generation — transformers 4.2.0 documentation
2021-02-03T11:29:23Z
I will definitely look into it! thank you again