Llama can now see and run on your device - welcome Llama 3.2
merve, philschmid, osanseviero, reach-vb, lewtun, ariG23498, pcuenq
Llama 3.2 is out! Today, we welcome the next iteration of the [Llama collection](https://huggingface.co/collections/meta-llama/llama-32-66f448ffc8c32f949b04c8cf) to Hugging Face. This time, we’re excited to collaborate with Meta on the release of multimodal and small models. Ten open-weight models (5 multimodal models and 5 text-only ones) are available on the Hub.

Llama 3.2 Vision comes in two sizes: 11B for efficient deployment and development on consumer-size GPU, and 90B for large-scale applications. Both versions come in base and instruction-tuned variants. In addition to the four multimodal models, Meta released a new version of Llama Guard with vision support. Llama Guard 3 is a safeguard model that can classify model inputs and generations, including detecting harmful multimodal prompts or assistant responses.

Llama 3.2 also includes small text-only language models that can run on-device. They come in two new sizes (1B and 3B) with base and instruct variants, and they have strong capabilities for their sizes. There’s also a small 1B version of Llama Guard that can be deployed alongside these or the larger text models in production use cases.

Among the features and integrations being released, we have:
- [Model checkpoints on the Hub](https://huggingface.co/collections/meta-llama/llama-32-66f448ffc8c32f949b04c8cf)
- [Hugging Face Transformers](https://huggingface.co/docs/transformers/v4.45.1/en/model_doc/mllama) and TGI integration for the Vision models
- Inference & Deployment Integration with Inference Endpoints, Google Cloud, Amazon SageMaker & DELL Enterprise Hub
- Fine-tuning Llama 3.2 11B Vision on a single GPU with [transformers🤗](https://github.com/huggingface/huggingface-llama-recipes/blob/main/fine_tune/Llama-Vision%20FT.ipynb) and [TRL](https://github.com/huggingface/huggingface-llama-recipes/blob/main/fine_tune/sft_vlm.py)

## Table of contents

- [What is Llama 3.2 Vision?](#what-is-llama-32-vision)
- [Llama 3.2 license changes. Sorry, EU :(](#llama-32-license-changes-sorry-eu-)
- [What is special about Llama 3.2 1B and 3B?](#what-is-special-about-llama-32-1b-and-3b)
- [Demo](#demo)
- [Using Hugging Face Transformers](#using-hugging-face-transformers)
- [Llama 3.2 1B & 3B Language Models](#llama-32-1b--3b-language-models)
- [Llama 3.2 Vision](#llama-32-vision)
- [On-device](#on-device)
- [Llama.cpp & Llama-cpp-python](#llamacpp--llama-cpp-python)
- [Transformers.js](#transformersjs)
- [Fine-tuning Llama 3.2](#fine-tuning-llama-32)
- [Hugging Face Partner Integrations](#hugging-face-partner-integrations)
- [Additional Resources](#additional-resources)
- [Acknowledgements](#acknowledgements)

## What is Llama 3.2 Vision?

Llama 3.2 Vision is the most powerful open multimodal model released by Meta. It has great visual understanding and reasoning capabilities and can be used to accomplish a variety of tasks, including visual reasoning and grounding, document question answering, and image-text retrieval. Chain of Thought (CoT) answers are often very good, which makes visual reasoning particularly powerful.

Llama 3.2 Vision can be used to process text and an image as well as only text. With image-text prompting, the model can take English inputs, while for text-only prompting, the model can handle multiple languages.

The full list of languages in text-only mode is:

- English
- German
- French
- Italian
- Portuguese
- Hindi
- Spanish
- Thai

The architecture of these models is based on the combination of Llama 3.1 LLMs combined with a vision tower and an image adapter. The text models used are Llama 3.1 8B for the Llama 3.2 11B Vision model, and Llama 3.1 70B for the 3.2 90B Vision model. To the best of our understanding, the text models were frozen during the training of the vision models to preserve text-only performance.

Below you can find some inference examples from the 11B instruction-tuned model that showcase real world knowledge, document reasoning and infographics understanding capabilities.


<table><thead><tr><th>Input Image</th><th>Input Prompt</th><th>Answer</th></tr></thead><tbody><tr> 
  <tr>
    <td style="vertical-align: middle;"><img src="https://huggingface.co/datasets/huggingface/release-assets/resolve/main/wat_arun.jpg" width="300"/></td>
    <td style="vertical-align: middle;">
    <strong>Where is this located in? What are other attractions nearby?</strong>
    </td>
    <td style="vertical-align: middle;">
      The image depicts the Grand Palace in Bangkok, Thailand. The Grand Palace is a complex of buildings that served as the official residence of the King of Siam and the administrative seat of government for 150 years. It is one of the most popular tourist attractions in Bangkok and a must-see for anyone visiting the city.<br><br>
      Some other attractions near the Grand Palace include:<br>
      * Wat Phra Kaew: This temple is located within the Grand Palace complex and is home to the famous Emerald Buddha statue.<br>
      * Wat Arun: This temple is located on the west bank of the Chao Phraya River and is known for its stunning architecture and beautiful views of the city.<br>
      * Wat Pho: This temple is located near the Grand Palace and is home to a large reclining Buddha statue.<br>
    </td>
  </tr>
  <tr>
    <td style="vertical-align: middle;"><img src="https://huggingface.co/datasets/huggingface/release-assets/resolve/main/weather_events.png" width="300"/></td>
    <td style="vertical-align: middle;">
    <strong>Where do the droughts happen according to this diagram?</strong>
    </td>
    <td style="vertical-align: middle;">
      <strong>Drought Regions in 2016</strong><br>According to the diagram, droughts occurred in two distinct regions in 2016:<br>
      * <strong>Eastern and Southern Africa:</strong> This region experienced severe droughts.<br>
      * <strong>Chile:</strong> Chile was the driest in January in over 50 years. These areas faced significant water scarcity issues during that period.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: middle;"><img src="https://huggingface.co/datasets/huggingface/release-assets/resolve/main/invoice.png" width="300"/></td>
    <td style="vertical-align: middle;">
    <strong>How long does it take from invoice date to due date? Be short and concise.</strong>
    </td>
    <td style="vertical-align: middle;">
      <strong>Invoice Date:</strong> 11/02/2019<br>
      <strong>Due Date:</strong> 26/02/2019<br>
      <strong>Time Difference Calculation:</strong><br>
      26/02/2019 - 11/02/2019 = 15 days<br>
      The time difference between the invoice date and the due date is <strong>15 days.</strong>
    </td>
  </tr>
</table>
  

Vision models have a context length of 128k tokens, which allows for multiple-turn conversations that may contain images. However, the model works best when attending to a single image, so the `transformers` implementation only attends to the last image provided in the input. This preserves quality and saves memory.

The 11B base model supports tile sizes of 448, while the instruct version and the 90B models all use tile sizes of 560. The models were trained on a massive dataset of 6 billion image-text pairs with a diverse data mixture. This makes them excellent candidates for fine-tuning on downstream tasks. For reference, you can see below how the 11B, 90B and their instruction fine-tuned versions compare in some benchmarks, as reported by Meta. Please, refer to the model cards for additional benchmarks and details.

|            | 11B               | 11B (instruction-tuned) | 90B               | 90B (instruction-tuned) | Metric | 
|