Benchmarking Text Generation Inference
derek-thomas
In this blog we will be exploring [Text Generation Inference’s](https://github.com/huggingface/text-generation-inference) (TGI) little brother, the [TGI Benchmarking tool](https://github.com/huggingface/text-generation-inference/blob/main/benchmark/README.md). It will help us understand how to profile TGI beyond simple throughput to better understand the tradeoffs to make decisions on how to tune your deployment for your needs. If you have ever felt like LLM deployments cost too much or if you want to tune your deployment to improve performance this blog is for you!

I’ll show you how to do this in a convenient [Hugging Face Space](https://huggingface.co/spaces). You can take the results and use it on an [Inference Endpoint](https://huggingface.co/inference-endpoints/dedicated) or other copy of the same hardware.


## Motivation

To get a better understanding of the need to profile, let's discuss some background information first.

Large Language Models (LLMs) are fundamentally inefficient. Based on [the way decoders work](https://huggingface.co/learn/nlp-course/chapter1/6?fw=pt), generation requires a new forward pass for each decoded token. As LLMs increase in size, and [adoption rates surge](https://a16z.com/generative-ai-enterprise-2024/) across enterprises, the AI industry has done a great job of creating new optimizations and performance enhancing techniques.

There have been dozens of improvements in many aspects of serving LLMs. We have seen [Flash Attention](https://huggingface.co/docs/text-generation-inference/en/conceptual/flash_attention), [Paged Attention](https://huggingface.co/docs/text-generation-inference/en/conceptual/paged_attention), [streaming responses](https://huggingface.co/docs/text-generation-inference/en/conceptual/streaming), [improvements in batching](https://huggingface.co/docs/text-generation-inference/en/basic_tutorials/launcher#maxwaitingtokens), [speculation](https://huggingface.co/docs/text-generation-inference/en/conceptual/speculation), [quantization](https://huggingface.co/docs/text-generation-inference/en/conceptual/quantization) of many kinds, [improvements in web servers](https://github.com/huggingface/text-generation-inference?tab=readme-ov-file#architecture), adoptions of [faster languages](https://github.com/search?q=repo%3Ahuggingface%2Ftext-generation-inference++language%3ARust&type=code) (sorry python 🐍), and many more. There are also use-case improvements like [structured generation](https://huggingface.co/docs/text-generation-inference/en/conceptual/guidance) and [watermarking](https://huggingface.co/blog/watermarking) that now have a place in the LLM inference world. The problem is that fast and efficient implementations require more and more niche skills to implement [[1]](#1). 

[Text Generation Inference](https://github.com/huggingface/text-generation-inference) is a high-performance LLM inference server from Hugging Face designed to embrace and develop the latest techniques in improving the deployment and consumption of LLMs. Due to Hugging Face’s open-source partnerships, most (if not all) major Open Source LLMs are available in TGI on release day.

Oftentimes users will have very different needs depending on their use-case requirements. Consider prompt and generation in a **RAG use-case**: 
* Instructions/formatting
    * usually short, <200 tokens
* The user query
    * usually short, <200 tokens
* Multiple documents 
    * medium-sized, 500-1000 tokens per document, 
    * N documents where N<10
* An answer in the output 
    * medium-sized ~500-1000 tokens
 
In RAG it's important to have the right document to get a quality response, you increase this chance by increasing N which includes more documents. This means that RAG will often try to max out an LLM’s context window to increase task performance. In contrast, think about basic chat. Typical **chat scenarios** have significantly fewer tokens than RAG:
* Multiple turns
    * 2xTx50-200 tokens, for T turns
    * The 2x is for both User and Assistant 

Given that we have such different scenarios, we need to make sure that we configure our LLM server accordingly depending on which one is more relevant. Hugging Face has a [benchmarking tool](https://github.com/huggingface/text-generation-inference/blob/main/benchmark/README.md) that can help us explore what configurations make the most sense and I'll explain how you can do this on a [Hugging Face Space](https://huggingface.co/docs/hub/en/spaces-overview). 


## Pre-requisites

Let’s make sure we have a common understanding of a few key concepts before we dive into the tool.

### Latency vs Throughput
<video style="width: auto; height: auto;" controls autoplay muted loop>
  <source src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tgi-benchmarking/LatencyThroughputVisualization.webm" type="video/webm">
  Your browser does not support the video tag.
</video>

|                                                 |
|