Welcome PaliGemma 2 – New vision language models by Google
merve, andsteing, pcuenq, ariG23498
We are excited to welcome Google's all-new vision language models, PaliGemma 2, a new iteration of PaliGemma. Like its predecessor, PaliGemma 2 uses the same powerful [SigLIP](https://huggingface.co/collections/google/siglip-659d5e62f0ae1a57ae0e83ba) for vision, but it upgrades to the latest Gemma 2 for the text decoder part.

PaliGemma 2 comes with new pre-trained (pt) models, in sizes of `3B`, `10B`, and `28B` parameters. All of them support various input resolutions: `224x224`, `448x448`, and `896x896`. These combinations provide a lot of flexibility for different use cases, so practitioners can choose the balance they need in the quality / efficiency space. In contrast, the previous PaliGemma was only available in the 3B variant.

The pre-trained models have been designed for easy fine-tuning to downstream tasks. The first PaliGemma was widely adopted by the community for multiple purposes. With the increased flexibility from the additional variants, combined with better pre-trained quality, we can’t wait to see what the community can do this time.

As an example, Google is also releasing some fine-tuned variants on the [DOCCI](https://huggingface.co/datasets/google/docci) dataset, demonstrating versatile and robust captioning capabilities that are long, nuanced and detailed. The fine-tuned DOCCI models are available for the 3B and 10B variants, and support input resolution of 448x448.


This release includes all the open model repositories, transformers integration, fine-tuning scripts, and a demo of a model we fine-tuned ourselves for visual question answering on the [VQAv2 dataset](https://huggingface.co/datasets/HuggingFaceM4/VQAv2).


- [Release collection](https://huggingface.co/collections/google/paligemma-2-release-67500e1e1dbfdd4dee27ba48)
  
- [Fine-tuning Script](https://github.com/merveenoyan/smol-vision/blob/main/Fine_tune_PaliGemma.ipynb)
  
- [Demo for Fine-tuned Model](https://huggingface.co/spaces/merve/paligemma2-vqav2)
  
- [The technical report](https://huggingface.co/papers/2412.03555)


## Table of Content

* [Introducing PaliGemma 2](#introducing-paligemma-2)

* [Model Capabilities](#model-capabilities)

* [Demo](#demo)

* [How to Use with transformers](#how-to-use-with-transformers)

* [Fine-tuning](#fine-tuning)

* [Resources](#resources)

## Introducing PaliGemma 2

PaliGemma 2 is a new iteration of the [PaliGemma vision language model](https://huggingface.co/blog/paligemma) released by Google in May. PaliGemma 2 connects the powerful SigLIP image encoder with the [Gemma 2](https://huggingface.co/blog/gemma2) language model.

![PaliGemma2 Architecture](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/paligemma/paligemma2_arch.png)

The new models are based on the Gemma 2 2B, 9B, and 27B language models, resulting in the corresponding 3B, 10B, and 28B PaliGemma 2 variants, whose names take into account the additional parameters of the (compact) image encoder. As mentioned above, they support three different resolutions, providing great flexibility for fine-tuning to downstream tasks.

PaliGemma 2 is distributed under the Gemma license, which allows for redistribution, commercial use, fine-tuning and creation of model derivatives.

This release comes with the following checkpoints in `bfloat16` precision:

- 9 pre-trained models: 3B, 10B, and 28B with resolutions of `224x224`, `448x448`, and `896x896`.
  
- 2 models fine-tuned on DOCCI: Two models fine-tuned on the [DOCCI](https://huggingface.co/datasets/google/docci) dataset (image-text caption pairs), supporting the 3B and 10B PaliGemma 2 variants and input resolution of `448x448`.
  
## Model Capabilities

As seen with the previous PaliGemma release, the pre-trained (pt) models work great for further fine-tuning on downstream tasks. 

The pt models are pre-trained on the following data mixture. The diversity of the pre-training dataset allows fine-tuning on downstream tasks in similar domains to be carried out using comparatively fewer examples.

- **WebLI**: A web-scale multilingual image-text dataset built from the public web. A wide range of WebLI splits is used to acquire versatile model capabilities, such as visual semantic understanding, object localization, visually-situated text understanding, and multilinguality.
  
- **CC3M-35L:** Curated English image-alt_text pairs from webpages ([Sharma et al., 2018](https://aclanthology.org/P18-1238/)). To label this dataset, the authors used [Google Cloud Translation API](https://cloud.google.com/translate) to translate into 34 additional languages.
  
- **Visual Question Generation with Question Answering Validation (VQ2A):** An improved dataset for question answering. The dataset is translated into the same additional 34 languages, using the Google Cloud Translation API.
  
OpenImages: Detection and object-aware questions and answers (Piergiovanni et al. 2022) generated by handcrafted rules on the [OpenImages dataset](https://storage.googleapis.com/openimages/web/factsfigures_v7.html).
- **WIT**: Images and texts collected from Wikipedia (Srinivasan et al., 2021). 

The PaliGemma 2 team internally fine-tuned the PT models on a wide variety of visual-language understanding tasks, and they provide benchmarks of these fine-tuned models [in the model card](https://huggingface.co/google/paligemma2-28b-pt-896#paligemma-2-results-by-model-resolution-and-size) and [the technical report](https://huggingface.co/papers/2412.03555).

PaliGemma 2 fine-tuned on the DOCCI dataset, can accomplish a wide range of captioning tasks, including text rendering, capturing spatial relations, and including world knowledge in captions.

You can find below the performance of the DOCCI fine-tuned PaliGemma 2 checkpoints, compared with other models (taken from Table 6 in [the technical report](https://huggingface.co/papers/2412.03555)).

|                 | #par    | #char   | #sent   | NES↓     |
|