SmolVLM - small yet mighty Vision Language Model
andito, merve, mfarre, eliebak, pcuenq
This blog post introduces SmolVLM, a 2B VLM, SOTA for its memory footprint. SmolVLM is small, fast, memory-efficient, and fully open-source. All model checkpoints, VLM datasets, training recipes and tools are released under the Apache 2.0 license.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/smolvlm_ecosystem.png" width="800" height="auto" alt="Image description">


## What is SmolVLM?

This year has seen a boom in multimodal AI with many large vision language models released. The trends were to initially scale up compute, later scale up the data diversity by generating synthetic data with large models, and, recently, scale down to make these models more efficient. Small open models allow local deployment to browser or edge devices, cut inference costs, and enable user customization. Some notable examples of these models include PaliGemma 3B, moondream2, and Qwen2VL.

In this blog post, we introduce [SmolVLM](https://huggingface.co/HuggingFaceTB/SmolVLM-Instruct), a new family of 2B small vision language models that can be used commercially and deployed to smaller local setups, with completely open training pipelines. 

We release three models: [SmolVLM-Base](https://huggingface.co/HuggingFaceTB/SmolVLM-Base), which can be used for downstream fine-tuning, [SmolVLM-Synthetic](https://huggingface.co/HuggingFaceTB/SmolVLM-Synthetic), the fine-tuned variant on synthetic data, and [SmolVLM Instruct](https://huggingface.co/HuggingFaceTB/SmolVLM-Instruct), the fine-tuned instruction variant, which can be used out of the box for interactive end-user applications. 

This release comes with open-source models integrated into transformers, [a demo built on SmolVLM Instruct](https://huggingface.co/spaces/HuggingFaceTB/SmolVLM), and a supervised fine-tuning script. We have used the datasets previously used for Idefics3: [the Cauldron](https://huggingface.co/datasets/HuggingFaceM4/the_cauldron) and [Docmatix](https://huggingface.co/datasets/HuggingFaceM4/Docmatix), which are also fully open-source.



## Table of Contents
- [TLDR](#TLDR)
- [What is SmolVLM?](#what-is-smolvlm)
  * [Model capabilities](#model-capabilities)
  * [Architecture](#Architecture)
- [Performance](#performance)
  * [Benchmarks](#benchmarks)
  * [Memory](#memory)
  * [Throughput](#throughput)
  * [Video](#video)
  * [VLMEvalKit integration](#VLMEvalKit)
- [Use SmolVLM with Transformers](#use-smolvlm-with-transformers)
- [Training Details](#training-details)
  * [Dataset](#dataset)
  * [Context extension](#context-extension)
  * [Checkpoint selection](#checkpoint-selection)
- [Fine-Tuning](#fine-tuning)
- [Wrapping up](#wrapping-up) 


## Model capabilities

<table><thead><tr><th>Input Image</th><th>Input Prompt</th><th>Answer</th></tr></thead><tbody><tr> 
  <tr>
    <td style="vertical-align: middle;"><img src="https://huggingface.co/datasets/huggingface/release-assets/resolve/main/wat_arun.jpg" width="300"/></td>
    <td style="vertical-align: middle;">
    <strong>I'm planning a visit to this temple, give me travel tips.</strong>
    </td>
    <td style="vertical-align: middle;">
      The Grand Palace is the most famous landmark in Bangkok and it's worth visiting for its stunning architecture and history. Here are some of our favorite things about the palace: 

**The Main Building:** 
Located at the center of the complex, the main building features intricate tile work with traditional Thai motifs that cover every inch of the structure. This includes patterns such as dragons, mythical creatures, flowers, and geometric shapes. The roofline has an elaborate design featuring multiple layers of tiles adorned with gold accents. 

**Towers & Spires:** 
There are several towers within the Grand Palace including the Phra Nakhon Si Ayutthaya National Museum Tower which houses exhibits on Thai culture and heritage. Another notable tower is the Wat Yai Chai Mongkol Tower where you can find beautiful murals depicting scenes from Buddhist mythology.

**Architecture Style:** 
The architectural style combines elements of Hindu and Khmer styles seen commonly throughout Southeast Asia. You'll notice influences like stupas (dome-shaped structures), lotus flower carvings, and ornate details typical of these regions' religious buildings.

**Visitor Experience:** 
While exploring the Grand Palace, visitors should be aware of certain rules due to security measures taken by authorities. These include no photography inside certain areas or using flashbulbs outside. Additionally, there may be queues during peak tourist seasons so plan accordingly if you're traveling here.

### Travel Tips For Visiting The Grand Palace:
* **Best Time To Visit**: The best time to visit would likely be early morning before sunrise when temperatures drop significantly making walking around easier.
* **Transportation**: Public transportation options exist but they might not provide direct access to the Grand Palace itself; consider taking a taxi instead.
* **Food Options**: There aren't many dining facilities available near the Grand Palace yet local street food vendors do offer delicious meals nearby.
* **Clothing Guidelines**: While not mandatory, wearing traditional Thai clothing (sarongs) could add cultural immersion experience. However, shorts/trousers are allowed provided they don't go below knee level.
* **Safety Measures**: Visitors must follow signs indicating restricted zones marked "No Photography" among others. Always carry identification documents just in case.

By following these guidelines, your trip will undoubtedly enhance both your understanding of Thailand’s rich historical tapestry and personal enjoyment!
    </td>
  </tr>
  <tr>
    <td style="vertical-align: middle;"><img src="https://huggingface.co/datasets/huggingface/release-assets/resolve/main/weather_events.png" width="300"/></td>
    <td style="vertical-align: middle;">
    <strong>Where do the severe droughts happen according to this diagram?</strong>
    </td>
    <td style="vertical-align: middle;">
      The severe droughts are happening in eastern and southern Africa.
    </td>
  </tr>
  <tr>
    <td style="vertical-align: middle;"><img src="https://huggingface.co/datasets/huggingface/release-assets/resolve/main/invoice.png" width="300"/></td>
    <td style="vertical-align: middle;">
    <strong>What is the due date and the invoice date?</strong>
    </td>
    <td style="vertical-align: middle;">
 The due date is 26/02/2019, and the invoice date is 11/02/2019.
    </td>
  </tr>
</table>

## Architecture

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/self_attention_architecture_smolvlm.png" width="800" height="auto" alt="Image description">

For SmolVLM, we closely followed the architecture from Idefics3, to the point that we use the same implementation in transformers. There are, however a few key differences:
 - We replaced Llama 3.1 8B with SmolLM2 1.7B as the language backbone.
 - We more aggressively compress the patched visual information by reducing the information 9x using the pixel shuffle strategy, compared to 4x with idefics3.
 - We use patches of 384*384, instead of 364x364, because 384 is divisible by 3, which is necessary for our pixel shuffle strategy to work.
 - For this, we change the vision backbone to use shape-optimized SigLIP with patches of 384x384 pixels and inner patches of 14x14.

## Performance

### Benchmarks

We present benchmarks for the tasks we mention in training details.
| Model             | MMMU (val) | MathVista (testmini) | MMStar (val) | DocVQA (test) | TextVQA (val) | Min GPU RAM required (GB) |
|