Falcon 2: An 11B parameter pretrained language model and VLM, trained on over 5000B tokens and 11 languages
Quent-01, nilabhra, rcojocaru, Mughaira, gcamp, yasserTII, SanathNarayan, griffintaur, clefourrier, SailorTwift
<a name="the-falcon-models"></a>
## The Falcon 2 Models

[TII](www.tii.ae) is launching a new generation of models, [Falcon 2](https://falconllm.tii.ae/), focused on providing the open-source community with a series of smaller models with enhanced performance and multi-modal support. Our goal is to enable cheaper inference and encourage the development of more downstream applications with improved usability.

The first generation of Falcon models, featuring [Falcon-40B](https://huggingface.co/tiiuae/falcon-40b) and [Falcon-180B](https://huggingface.co/tiiuae/falcon-180B), made a significant contribution to the open-source community, promoting the release of advanced LLMs with permissive licenses. More detailed information on the previous generation of Falcon models can be found in the [RefinedWeb, Penedo et al., 2023](https://proceedings.neurips.cc/paper_files/paper/2023/hash/fa3ed726cc5073b9c31e3e49a807789c-Abstract-Datasets_and_Benchmarks.html) and [The Falcon Series of Open Language Models, Almazrouei et al., 2023](https://arxiv.org/abs/2311.16867) papers, and the [Falcon](https://huggingface.co/blog/falcon) and [Falcon-180B](https://huggingface.co/blog/falcon-180b) blog posts.

The second generation of models is focused on increased usability and integrability, building a multi-modal ecosystem. We start this journey by releasing not only the base [11B LLM](https://huggingface.co/tiiuae/falcon-11B), but also the [11B VLM model](https://huggingface.co/tiiuae/Falcon-11B-vlm) that incorporates image understanding capabilities. The vision-language model, or VLM, will allow users to engage in chats about visual content using text.

As with our previous work, the models offer support mainly in English but have good capabilities in ten other languages, including Spanish, French, and German.

## Table of Contents

- [The Falcon 2 Models](#the-falcon-models)
- Falcon 2 11B LLM
    - [11B LLM Training Details](#falcon2-11b-llm)
    - [11B LLM Evaluation](#falcon2-11b-evaluation)
    - [11B LLM Using the Model](#using-falcon2-11b)
- Falcon 2 11B VLM
    - [11B VLM Training](#falcon2-11b-vlm)
    - [11B VLM Evaluation](#falcon2-11b-vlm-evaluation)
    - [11B VLM Using the Model](#using-falcon2-11b-falconvlm)
- [Licensing information](#license-information)

<a name="falcon2-11b-llm"></a>
## Falcon2-11B LLM

### Training Data
Falcon2-11B was trained on over 5,000 GT (billion tokens) of RefinedWeb, a high-quality filtered and deduplicated web dataset, enhanced with curated corpora. It followed a four-stage training strategy. The first three stages were focused on increasing the context length, from 2048 to 4096 and finally to 8192 tokens. The last stage aimed to further enhance performance using only high-quality data.

Overall, the data sources included RefinedWeb-English, RefinedWeb-Europe (*cs*, *de*, *es*, *fr*, *it*, *nl*, *pl*, *pt*, *ro*, *sv*), high-quality technical data, code data, and conversational data extracted from public sources.

The training stages were as follows:

| Stage   | Context Length   | GT   | 
|