Add files using upload-large-folder tool

README.md

@@ -1,59 +1,22 @@
 ---
-base_model: Qwen/Qwen2.5-VL-72B-Instruct
+license: other
+license_name: qwen
+license_link: https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct/blob/main/LICENSE
 language:
 - en
-library_name: transformers
 pipeline_tag: image-text-to-text
-license: apache-2.0
 tags:
 - multimodal
-- qwen
-- qwen2
 - unsloth
-- transformers
-- vision
+library_name: transformers
+base_model:
+- Qwen/Qwen2.5-VL-72B-Instruct
 ---
-<div>
-    <p style="margin-bottom: 0;margin-top:0;">
-    <em>View all of our uploaded models <a href="https://docs.unsloth.ai/get-started/all-our-models">here</em>
-  </p>
-  <div style="display: flex; gap: 5px; align-items: center;margin-top:0; ">
-    <a href="https://github.com/unslothai/unsloth/">
-      <img src="https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png" width="133">
-    </a>
-    <a href="https://discord.gg/unsloth">
-      <img src="https://github.com/unslothai/unsloth/raw/main/images/Discord%20button.png" width="173">
-    </a>
-    <a href="https://docs.unsloth.ai/">
-      <img src="https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/images/documentation%20green%20button.png" width="143">
-    </a>
-  </div>
-<h1 style="margin-top: 0rem;">Finetune LLMs 2-5x faster with 70% less memory via Unsloth</h2>
-</div>
-We have a free Google Colab Tesla T4 notebook for Qwen2-VL (7B) here: https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2_VL_(7B)-Vision.ipynb
-
-## ✨ Finetune for Free
-
-All notebooks are **beginner friendly**! Add your dataset, click "Run All", and you'll get a 2x faster finetuned model which can be exported to GGUF, vLLM or uploaded to Hugging Face.
-
-| Unsloth supports          |    Free Notebooks                                                                                           | Performance | Memory use |
-|-----------------|--------------------------------------------------------------------------------------------------------------------------|-------------|----------|
-| **Llama-3.2 (3B)**      | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(1B_and_3B)-Conversational.ipynb)               | 2.4x faster | 58% less |
-| **Llama-3.2 (11B vision)**      | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb)               | 2x faster | 60% less |
-| **Qwen2 VL (7B)**      | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2_VL_(7B)-Vision.ipynb)               | 1.8x faster | 60% less |
-| **Qwen2.5 (7B)**      | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Qwen2.5_(7B)-Alpaca.ipynb)               | 2x faster | 60% less |
-| **Llama-3.1 (8B)**      | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-Alpaca.ipynb)               | 2.4x faster | 58% less |
-| **Phi-3.5 (mini)** | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Phi_3.5_Mini-Conversational.ipynb)               | 2x faster | 50% less |
-| **Gemma 2 (9B)**      | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma2_(9B)-Alpaca.ipynb)               | 2.4x faster | 58% less |
-| **Mistral (7B)**    | [▶️ Start on Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_v0.3_(7B)-Conversational.ipynb)               | 2.2x faster | 62% less |
-
-[<img src="https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/images/documentation%20green%20button.png" width="200"/>](https://docs.unsloth.ai)
-
-- This [Llama 3.2 conversational notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(1B_and_3B)-Conversational.ipynb) is useful for ShareGPT ChatML / Vicuna templates.
-- This [text completion notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Mistral_(7B)-Text_Completion.ipynb) is for raw text. This [DPO notebook](https://colab.research.google.com/drive/15vttTpzzVXv_tJwEk-hIcQ0S9FcEWvwP?usp=sharing) replicates Zephyr.
-- \* Kaggle has 2x T4s, but we use 1. Due to overhead, 1x T4 is 5x faster.
-
-# Qwen2.5-VL
+
+# Qwen2.5-VL-72B-Instruct
+<a href="https://chat.qwenlm.ai/" target="_blank" style="margin: 2px;">
+    <img alt="Chat" src="https://img.shields.io/badge/%F0%9F%92%9C%EF%B8%8F%20Qwen%20Chat%20-536af5" style="display: inline-block; vertical-align: middle;"/>
+</a>
 
 ## Introduction
 
@@ -81,13 +44,12 @@ We extend dynamic resolution to the temporal dimension by adopting dynamic FPS s
     <img src="https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen2.5-VL/qwen2.5vl_arc.jpeg" width="80%"/>
 <p>
 
-
 * **Streamlined and Efficient Vision Encoder**
 
 We enhance both training and inference speeds by strategically implementing window attention into the ViT. The ViT architecture is further optimized with SwiGLU and RMSNorm, aligning it with the structure of the Qwen2.5 LLM.
 
 
-We have three models with 3, 7 and 72 billion parameters. This repo contains the instruction-tuned 7B Qwen2.5-VL model. For more information, visit our [Blog](https://qwenlm.github.io/blog/qwen2.5-vl/) and [GitHub](https://github.com/QwenLM/Qwen2.5-VL).
+We have three models with 3, 7 and 72 billion parameters. This repo contains the instruction-tuned 72B Qwen2.5-VL model. For more information, visit our [Blog](https://qwenlm.github.io/blog/qwen2.5-vl/) and [GitHub](https://github.com/QwenLM/Qwen2.5-VL).
 
 
 
@@ -95,50 +57,51 @@ We have three models with 3, 7 and 72 billion parameters. This repo contains the
 
 ### Image benchmark
 
+|      Benchmarks        | GPT4o     | Claude3.5 Sonnet  | Gemini-2-flash  | InternVL2.5-78B | Qwen2-VL-72B | Qwen2.5-VL-72B |
+|-----------------------|-----------|-------------------|-----------------|-----------------|--------------|----------------|
+| MMMU<sub>val</sub>    | 70.3      | 70.4              | 70.7            | 70.1                 | 64.5         | 70.2          |
+| MMMU_Pro              | 54.5      | 54.7              | 57.0            | 48.6              | 46.2         | 51.1           |
+| MathVista_MINI        | 63.8      | 65.4              | 73.1            | 76.6                | 70.5         | 74.8           |
+| MathVision_FULL       | 30.4      | 38.3              | 41.3            | 32.2               | 25.9         | 38.1           |
+| Hallusion Bench       | 55.0      | 55.16             |                | 57.4             | 58.1         | 55.16           |
+| MMBench_DEV_EN_V11    | 82.1      | 83.4              | 83.0            | 88.5             | 86.6         | 88           |
+| AI2D_TEST             | 84.6      | 81.2              |                 | 89.1           | 88.1         | 88.4           |
+| ChartQA_TEST          | 86.7      | 90.8              | 85.2            | 88.3               | 88.3         | 89.5           |
+| DocVQA_VAL            | 91.1      | 95.2              | 92.1            | 96.5             | 96.1         |      96.4      |
+| MMStar                | 64.7      | 65.1              | 69.4            | 69.5             | 68.3         |       70.8         |
+| MMVet_turbo           | 69.1      |  70.1              |                 | 72.3           | 74.0         |       76.19         |
+| OCRBench              | 736       | 788               |                 | 854               | 877          |         885       |
+| OCRBench-V2(en/zh)    |  46.5/32.3 |  45.2/39.6         | 51.9/43.1       | 45/46.2     | 47.8/46.1    | 61.5/63.7    |
+| CC-OCR                | 66.6     | 62.7              | 73.0            | 64.7          | 68.7       |79.8           |
+
+
+### Video benchmark
+| Benchmarks          | GPT4o | Gemini-1.5-Pro | InternVL2.5-78B | Qwen2VL-72B | Qwen2.5VL-72B |
+|---------------------|-------|----------------|-----------------|-------------|---------------|
+| VideoMME w/o sub.   | 71.9  | 75.0           | 72.1            | 71.2        | 73.3          |
+| VideoMME w sub.     | 77.2  | 81.3           | 74.0            | 77.8        | 79.1          |
+| MVBench             | 64.6  | 60.5           | 76.4            | 73.6        | 70.4          |
+| MMBench-Video       | 1.63  | 1.30           | 1.97            | 1.70        | 2.02          |
+| LVBench             | 30.8  | 33.1           | -               | 41.3        | 47.3          |
+| EgoSchema           | 72.2  | 71.2           | -               | 77.9        | 76.2          |
+| PerceptionTest_test | -     | -              | -               | 68.0        | 73.2          |
+| MLVU_M-Avg_dev      | 64.6  | -              | 75.7            |             | 74.6          |
+| TempCompass_overall | 73.8  | -              | -               |             | 74.8          |
 
-| Benchmark | InternVL2.5-8B | MiniCPM-o 2.6 | GPT-4o-mini | Qwen2-VL-7B |**Qwen2.5-VL-7B** |
-| :--- | :---: | :---: | :---: | :---: | :---: |
-| MMMU<sub>val</sub>  | 56 | 50.4 | **60**| 54.1 | 58.6|
-| MMMU-Pro<sub>val</sub>  | 34.3 | - | 37.6| 30.5 | 41.0|
-| DocVQA<sub>test</sub>  | 93 | 93 | - | 94.5 | **95.7** |
-| InfoVQA<sub>test</sub>  | 77.6 | - |  - |76.5 | **82.6** |
-| ChartQA<sub>test</sub>  | 84.8 | - |- | 83.0 |**87.3** |
-| TextVQA<sub>val</sub>  | 79.1 | 80.1 | -| 84.3 | **84.9**|
-| OCRBench | 822 | 852 | 785 | 845 | **864** |
-| CC_OCR | 57.7 |  | | 61.6 | **77.8**|
-| MMStar | 62.8| | |60.7| **63.9**|
-| MMBench-V1.1-En<sub>test</sub>  | 79.4 | 78.0 | 76.0| 80.7 | **82.6** |
-| MMT-Bench<sub>test</sub> | - | - | - |**63.7** |63.6 |
-| MMStar | **61.5** | 57.5 |  54.8 | 60.7 |63.9 |
-| MMVet<sub>GPT-4-Turbo</sub>  | 54.2 | 60.0 | 66.9 | 62.0 | **67.1**|
-| HallBench<sub>avg</sub>  | 45.2 | 48.1 | 46.1| 50.6 | **52.9**|
-| MathVista<sub>testmini</sub>  | 58.3 | 60.6 | 52.4 | 58.2 | **68.2**|
-| MathVision  | - | -  | - | 16.3 | **25.07** |
-
-### Video Benchmarks
-
-| Benchmark |  Qwen2-VL-7B | **Qwen2.5-VL-7B** |
-| :--- | :---: | :---: |
-| MVBench |  67.0 | **69.6** |
-| PerceptionTest<sub>test</sub>  | 66.9 | **70.5** |
-| Video-MME<sub>wo/w subs</sub>   | 63.3/69.0 | **65.1**/**71.6** |
-| LVBench  |  | 45.3 |
-| LongVideoBench  |  | 54.7 |
-| MMBench-Video | 1.44 | 1.79 |
-| TempCompass |  | 71.7 |
-| MLVU |  | 70.2 |
-| CharadesSTA/mIoU |  43.6|
 
 ### Agent benchmark
-| Benchmarks              | Qwen2.5-VL-7B |
-|-------------------------|---------------|
-| ScreenSpot              |     84.7    |
-| ScreenSpot Pro          |     29.0    |
-| AITZ_EM                 |  	81.9    |
-| Android Control High_EM |    	60.1    |
-| Android Control Low_EM  |  	93.7    |
-| AndroidWorld_SR         | 	25.5  	|
-| MobileMiniWob++_SR      | 	91.4    |
+
+| Benchmarks              | GPT4o       | Gemini 2.0 | Claude | Aguvis-72B | Qwen2VL-72B | Qwen2.5VL-72B |
+|-------------------------|-------------|------------|--------|------------|-------------|---------------|
+| ScreenSpot              | 18.1        | 84.0       | 83.0   |            |             | 87.1          |
+| ScreenSpot Pro          |             |            | 17.1   |            | 1.6         | 43.6          |
+| AITZ_EM                 | 35.3        |            |        |            | 72.8        | 83.2          |
+| Android Control High_EM |             |            |        | 66.4       | 59.1        | 67.36         |
+| Android Control Low_EM  |             |            |        | 84.4       | 59.2        | 93.7          |
+| AndroidWorld_SR         | 34.5% (SoM) |            | 27.9%  | 26.1%      |             | 35%           |
+| MobileMiniWob++_SR      |             |            |        | 66%        |             | 68%           |
+| OSWorld                 |             |            | 14.90  | 10.26      |             | 8.83          |
+
 
 ## Requirements
 The code of Qwen2.5-VL has been in the latest Hugging face transformers and we advise you to build from source with command:
@@ -184,25 +147,25 @@ from qwen_vl_utils import process_vision_info
 
 # default: Load the model on the available device(s)
 model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
-    "Qwen/Qwen2.5-VL-7B-Instruct", torch_dtype="auto", device_map="auto"
+    "Qwen/Qwen2.5-VL-72B-Instruct", torch_dtype="auto", device_map="auto"
 )
 
 # We recommend enabling flash_attention_2 for better acceleration and memory saving, especially in multi-image and video scenarios.
 # model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
-#     "Qwen/Qwen2.5-VL-7B-Instruct",
+#     "Qwen/Qwen2.5-VL-72B-Instruct",
 #     torch_dtype=torch.bfloat16,
 #     attn_implementation="flash_attention_2",
 #     device_map="auto",
 # )
 
 # default processer
-processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")
+processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-72B-Instruct")
 
 # The default range for the number of visual tokens per image in the model is 4-16384.
 # You can set min_pixels and max_pixels according to your needs, such as a token range of 256-1280, to balance performance and cost.
 # min_pixels = 256*28*28
 # max_pixels = 1280*28*28
-# processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct", min_pixels=min_pixels, max_pixels=max_pixels)
+# processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-72B-Instruct", min_pixels=min_pixels, max_pixels=max_pixels)
 
 messages = [
     {
@@ -471,7 +434,7 @@ The model supports a wide range of resolution inputs. By default, it uses the na
 min_pixels = 256 * 28 * 28
 max_pixels = 1280 * 28 * 28
 processor = AutoProcessor.from_pretrained(
-    "Qwen/Qwen2.5-VL-7B-Instruct", min_pixels=min_pixels, max_pixels=max_pixels
+    "Qwen/Qwen2.5-VL-72B-Instruct", min_pixels=min_pixels, max_pixels=max_pixels
 )
 ```
 
@@ -521,6 +484,7 @@ To handle extensive inputs exceeding 32,768 tokens, we utilize [YaRN](https://ar
 
 For supported frameworks, you could add the following to `config.json` to enable YaRN:
 
+```json
 {
 	...,
     "type": "yarn",
@@ -532,6 +496,7 @@ For supported frameworks, you could add the following to `config.json` to enable
     "factor": 4,
     "original_max_position_embeddings": 32768
 }
+```
 
 However, it should be noted that this method has a significant impact on the performance of temporal and spatial localization tasks, and is therefore not recommended for use.
 
@@ -539,7 +504,6 @@ At the same time, for long video inputs, since MRoPE itself is more economical w
 
 
 
-
 ## Citation
 
 If you find our work helpful, feel free to give us a cite.
@@ -567,4 +531,3 @@ If you find our work helpful, feel free to give us a cite.
   year={2023}
 }
 ```
-

chat_template.jinja

@@ -0,0 +1,7 @@
+{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system
+You are a helpful assistant.<|im_end|>
+{% endif %}<|im_start|>{{ message['role'] }}
+{% if message['content'] is string %}{{ message['content'] }}<|im_end|>
+{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>
+{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant
+{% endif %}

config.json

@@ -1,5 +1,4 @@
 {
-  "_name_or_path": "Qwen/Qwen2.5-VL-72B-Instruct",
   "architectures": [
     "Qwen2_5_VLForConditionalGeneration"
   ],
@@ -10,7 +9,7 @@
   "image_token_id": 151655,
   "initializer_range": 0.02,
   "intermediate_size": 29568,
-  "max_position_embeddings": 32768,
+  "max_position_embeddings": 128000,
   "max_window_layers": 80,
   "model_type": "qwen2_5_vl",
   "num_attention_heads": 64,
@@ -29,22 +28,76 @@
   },
   "rope_theta": 1000000.0,
   "sliding_window": 32768,
+  "text_config": {
+    "architectures": [
+      "Qwen2_5_VLForConditionalGeneration"
+    ],
+    "attention_dropout": 0.0,
+    "bos_token_id": 151643,
+    "eos_token_id": 151645,
+    "hidden_act": "silu",
+    "hidden_size": 8192,
+    "image_token_id": null,
+    "initializer_range": 0.02,
+    "intermediate_size": 29568,
+    "max_position_embeddings": 128000,
+    "max_window_layers": 80,
+    "model_type": "qwen2_5_vl_text",
+    "num_attention_heads": 64,
+    "num_hidden_layers": 80,
+    "num_key_value_heads": 8,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": {
+      "mrope_section": [
+        16,
+        24,
+        24
+      ],
+      "rope_type": "default",
+      "type": "default"
+    },
+    "rope_theta": 1000000.0,
+    "sliding_window": 32768,
+    "torch_dtype": "bfloat16",
+    "use_cache": true,
+    "use_sliding_window": false,
+    "video_token_id": null,
+    "vision_end_token_id": 151653,
+    "vision_start_token_id": 151652,
+    "vision_token_id": 151654,
+    "vocab_size": 152064
+  },
   "tie_word_embeddings": false,
   "torch_dtype": "bfloat16",
-  "transformers_version": "4.49.0",
+  "transformers_version": "4.52.0.dev0",
   "unsloth_fixed": true,
   "use_cache": true,
   "use_sliding_window": false,
   "video_token_id": 151656,
   "vision_config": {
+    "depth": 32,
+    "fullatt_block_indexes": [
+      7,
+      15,
+      23,
+      31
+    ],
+    "hidden_act": "silu",
     "hidden_size": 1280,
+    "in_channels": 3,
     "in_chans": 3,
+    "initializer_range": 0.02,
     "intermediate_size": 3456,
     "model_type": "qwen2_5_vl",
+    "num_heads": 16,
     "out_hidden_size": 8192,
+    "patch_size": 14,
+    "spatial_merge_size": 2,
     "spatial_patch_size": 14,
+    "temporal_patch_size": 2,
     "tokens_per_second": 2,
-    "torch_dtype": "bfloat16"
+    "torch_dtype": "bfloat16",
+    "window_size": 112
   },
   "vision_end_token_id": 151653,
   "vision_start_token_id": 151652,

generation_config.json

@@ -5,10 +5,9 @@
     151645,
     151643
   ],
-  "max_length": 32768,
+  "max_length": 128000,
   "pad_token_id": 151654,
   "repetition_penalty": 1.05,
-  "top_k": 1,
-  "top_p": 0.001,
-  "transformers_version": "4.49.0"
+  "temperature": 1e-06,
+  "transformers_version": "4.52.0.dev0"
 }

tokenizer_config.json

@@ -195,16 +195,16 @@
     "<|video_pad|>"
   ],
   "bos_token": null,
-  "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}",
   "clean_up_tokenization_spaces": false,
   "eos_token": "<|im_end|>",
   "errors": "replace",
   "extra_special_tokens": {},
-  "model_max_length": 32768,
+  "model_max_length": 128000,
   "pad_token": "<|vision_pad|>",
   "padding_side": "left",
   "processor_class": "Qwen2_5_VLProcessor",
   "split_special_tokens": false,
   "tokenizer_class": "Qwen2Tokenizer",
-  "unk_token": null
-}
+  "unk_token": null,
+  "chat_template": "{% set image_count = namespace(value=0) %}{% set video_count = namespace(value=0) %}{% for message in messages %}{% if loop.first and message['role'] != 'system' %}<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n{% endif %}<|im_start|>{{ message['role'] }}\n{% if message['content'] is string %}{{ message['content'] }}<|im_end|>\n{% else %}{% for content in message['content'] %}{% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}{% set image_count.value = image_count.value + 1 %}{% if add_vision_id %}Picture {{ image_count.value }}: {% endif %}<|vision_start|><|image_pad|><|vision_end|>{% elif content['type'] == 'video' or 'video' in content %}{% set video_count.value = video_count.value + 1 %}{% if add_vision_id %}Video {{ video_count.value }}: {% endif %}<|vision_start|><|video_pad|><|vision_end|>{% elif 'text' in content %}{{ content['text'] }}{% endif %}{% endfor %}<|im_end|>\n{% endif %}{% endfor %}{% if add_generation_prompt %}<|im_start|>assistant\n{% endif %}"
+}