How we sped up transformer inference 100x for 🤗 API customers
nan
🤗 Transformers has become the default library for data scientists all around the world to explore state of the art NLP models and build new NLP features. With over 5,000 pre-trained and fine-tuned models available, in over 250 languages, it is a rich playground, easily accessible whichever framework you are working in.

While experimenting with models in 🤗 Transformers is easy, deploying these large models into production with maximum performance, and managing them into an architecture that scales with usage is a **hard engineering challenge** for any Machine Learning Engineer. 

This 100x performance gain and built-in scalability is why subscribers of our hosted [Accelerated Inference API](https://huggingface.co/pricing) chose to build their NLP features on top of it. To get to the **last 10x of performance** boost, the optimizations need to be low-level, specific to the model, and to the target hardware.

This post shares some of our approaches squeezing every drop of compute juice for our customers. 🍋


## Getting to the first 10x speedup

The first leg of the optimization journey is the most accessible, all about using the best combination of techniques offered by the [Hugging Face libraries](https://github.com/huggingface/), independent of the target hardware. 

We use the most efficient methods built into Hugging Face model [pipelines](https://huggingface.co/transformers/main_classes/pipelines.html) to reduce the amount of computation during each forward pass. These methods are specific to the architecture of the model and the target task, for instance for a text-generation task on a GPT architecture, we reduce the dimensionality of the attention matrices computation by focusing on the new attention of the last token in each pass:

-| Naive version                                                                                             | Optimized version                                                                                       |
-|: