TGI Multi-LoRA: Deploy Once, Serve 30 Models
derek-thomas, dmaniloff, drbh
Are you tired of the complexity and expense of managing multiple AI models? **What if you could deploy once and serve 30 models?** In today's ML world, organizations looking to leverage the value of their data will likely end up in a _fine-tuned world_, building a multitude of models, each one highly specialized for a specific task. But how can you keep up with the hassle and cost of deploying a model for each use case? The answer is Multi-LoRA serving.


## Motivation

As an organization, building a multitude of models via fine-tuning makes sense for multiple reasons. 


- **Performance -** There is [compelling evidence](https://huggingface.co/papers/2405.09673) that smaller, specialized models outperform their larger, general-purpose counterparts on the tasks that they were trained on. Predibase [[5]](#5) showed that you can get better performance than GPT-4 using task-specific LoRAs with a base like [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1/tree/main).


- **Adaptability -** Models like Mistral or Llama are extremely versatile. You can pick one of them as your base model and build many specialized models, even when the [downstream tasks are very different](https://predibase.com/blog/lora-land-fine-tuned-open-source-llms-that-outperform-gpt-4). Also, note that you aren't locked in as you can easily swap that base and fine-tune it with your data on another base (more on this later).


- **Independence -** For each task that your organization cares about, different teams can work on different fine tunes, allowing for independence in data preparation,  configurations, evaluation criteria, and cadence of model updates.


- **Privacy -** Specialized models offer flexibility with training data segregation and access restrictions to different users based on data privacy requirements. Additionally, in cases where running models locally is important, a small model can be made highly capable for a specific task while keeping its size small enough to run on device. 

In summary, fine-tuning enables organizations to unlock the value of their data, and this advantage becomes especially significant, even game-changing, when organizations use highly specialized data that is uniquely theirs.

So, where is the catch? Deploying and serving Large Language Models (LLMs) is challenging in many ways. Cost and operational complexity are key considerations when deploying a single model, let alone _n_ models. This means that, for all its glory, fine-tuning complicates LLM deployment and serving even further.

That is why today we are super excited to introduce TGI's latest feature - **Multi-LoRA serving**.


## Background on LoRA

LoRA, which stands for [Low-Rank Adaptation](https://huggingface.co/papers/2106.09685), is a technique to fine-tune large pre-trained models efficiently. The core idea is to adapt large pre-trained models to specific tasks without needing to retrain the entire model, but only a small set of parameters called adapters. These adapters typically only add about 1% of storage and memory overhead compared to the size of the pre-trained LLM while maintaining the quality compared to fully fine-tuned models. 

The obvious benefit of LoRA is that it makes fine-tuning a lot cheaper by reducing memory needs. It also [reduces catastrophic forgetting](https://huggingface.co/papers/2405.09673) and works better with [small datasets](https://huggingface.co/blog/peft).

<video style="width: auto; height: auto;" controls autoplay muted loop>
  <source src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/multi-lora-serving/LoRA.webm">
  Your browser does not support the video tag.
</video>

|                            |
|