Update README.md

README.md

@@ -24,14 +24,16 @@ This repository contains the InfiGUI-G1-3B model from the paper **[InfiGUI-G1: A
   <a href="https://github.com/InfiXAI/InfiGUI-G1"><img src="https://img.shields.io/badge/GitHub-Repo-181717?style=flat&logo=github&logoColor=white" alt="GitHub Repo"></a>
 </p>
 
-## Paper Abstract
-
-The emergence of Multimodal Large Language Models (MLLMs) has propelled the development of autonomous agents that operate on Graphical User Interfaces (GUIs) using pure visual input. A fundamental challenge is robustly grounding natural language instructions. This requires a precise spatial alignment, which accurately locates the coordinates of each element, and, more critically, a correct semantic alignment, which matches the instructions to the functionally appropriate UI element. Although Reinforcement Learning with Verifiable Rewards (RLVR) has proven to be effective at improving spatial alignment for these MLLMs, we find that inefficient exploration bottlenecks semantic alignment, which prevent models from learning difficult semantic associations. To address this exploration problem, we present Adaptive Exploration Policy Optimization (AEPO), a new policy optimization framework. AEPO employs a multi-answer generation strategy to enforce broader exploration, which is then guided by a theoretically grounded Adaptive Exploration Reward (AER) function derived from first principles of efficiency eta=U/C. Our AEPO-trained models, InfiGUI-G1-3B and InfiGUI-G1-7B, establish new state-of-the-art results across multiple challenging GUI grounding benchmarks, achieving significant relative improvements of up to 9.0% against the naive RLVR baseline on benchmarks designed to test generalization and semantic understanding. Resources are available at this https URL .
-
 ## Model Description
 
 The model is based on `Qwen2.5-VL-3B-Instruct` and is fine-tuned using our proposed **Adaptive Exploration Policy Optimization (AEPO)** framework. AEPO is a novel reinforcement learning method designed to enhance the model's **semantic alignment** for GUI grounding tasks. It overcomes the exploration bottlenecks of standard RLVR methods by integrating a multi-answer generation strategy with a theoretically-grounded adaptive reward function, enabling more effective and efficient learning for complex GUI interactions.
 
+## Paper Overview
+
+A fundamental challenge for GUI agents is robustly grounding natural language instructions, which requires not only precise **spatial alignment** (locating elements accurately) but also correct **semantic alignment** (identifying the functionally appropriate element). While existing Reinforcement Learning with Verifiable Rewards (RLVR) methods have enhanced spatial precision, they often suffer from inefficient exploration. This "confidence trap" bottlenecks semantic alignment, preventing models from discovering correct actions for difficult semantic associations.
+
+To address this critical exploration problem, we introduce **InfiGUI-G1**, a series of models trained with **Adaptive Exploration Policy Optimization (AEPO)**. AEPO overcomes the exploration bottleneck by integrating a **multi-answer generation** strategy to explore a diverse set of candidate actions in a single forward pass. This exploration is guided by a theoretically-grounded **Adaptive Exploration Reward (AER)** function, derived from first principles of efficiency (η=U/C), which provides rich, informative learning signals to dynamically balance exploration and exploitation.
+
 ## Quick Start
 
 ### Installation
@@ -85,7 +87,7 @@ def load_image(img_path: str) -> Image.Image:
 
 
 def visualize_points(original_image: Image.Image, points: list, 
-                    new_width: int, new_height: int,\
+                    new_width: int, new_height: int,
                     original_width: int, original_height: int) -> None:
     """Draw prediction points on original image and save as output.png."""
     output_img = original_image.copy()
@@ -103,8 +105,8 @@ def visualize_points(original_image: Image.Image, points: list,
         
         # Draw circle
         circle_radius = 20
-        draw.ellipse([original_x - circle_radius, original_y - circle_radius,\
-                     original_x + circle_radius, original_y + circle_radius],\
+        draw.ellipse([original_x - circle_radius, original_y - circle_radius,
+                     original_x + circle_radius, original_y + circle_radius],
                     fill=(255, 0, 0))
         
         # Draw label
@@ -138,8 +140,7 @@ def main():
 
     # Prepare model inputs
     instruction = "shuffle play the current playlist"
-    system_prompt = 'You FIRST think about the reasoning process as an internal monologue and then provide the final answer.\
-The reasoning process MUST BE enclosed within <think> </think> tags.'
+    system_prompt = 'You FIRST think about the reasoning process as an internal monologue and then provide the final answer.\nThe reasoning process MUST BE enclosed within <think> </think> tags.'
     prompt = f'''The screen's resolution is {new_width}x{new_height}.
 Locate the UI element(s) for "{instruction}", output the coordinates using JSON format: [{{"point_2d": [x, y]}}, ...]'''
 
@@ -180,49 +181,77 @@ To reproduce the results in our paper, please refer to our repo for detailed ins
 
 ## Results
 
-Our InfiGUI-G1 models, trained with the AEPO framework, establish new state-of-the-art results among open-source models across a diverse and challenging set of GUI grounding benchmarks.
-
-### MMBench-GUI (L2) Results
-
-On the comprehensive MMBench-GUI benchmark, which evaluates performance across various platforms and instruction complexities, our InfiGUI-G1 models establish new state-of-the-art results for open-source models in their respective size categories.
-
-<div align="center">
-  <img src="https://raw.githubusercontent.com/InfiXAI/InfiGUI-G1/main/assets/results_mmbench-gui.png" width="90%" alt="MMBench-GUI Results">
-</div>
-
-### ScreenSpot-Pro Results
-
-On the challenging ScreenSpot-Pro benchmark, designed to test semantic understanding on high-resolution professional software, InfiGUI-G1 demonstrates significant improvements, particularly on icon-based grounding tasks. This highlights AEPO's effectiveness in enhancing semantic alignment by associating abstract visual symbols with their functions.
-
-<div align="center">
-  <img src="https://raw.githubusercontent.com/InfiXAI/InfiGUI-G1/main/assets/results_screenspot-pro.png" width="90%" alt="ScreenSpot-Pro Results">
-</div>
-
-### UI-Vision (Element Grounding) Results
-
-InfiGUI-G1 shows strong generalization capabilities on the UI-Vision benchmark, which is designed to test robustness across a wide variety of unseen desktop applications. Achieving high performance confirms that our AEPO framework fosters a robust understanding rather than overfitting to the training data.
-
-<div align="center">
-  <img src="https://raw.githubusercontent.com/InfiXAI/InfiGUI-G1/main/assets/results_ui-vision.png" width="90%" alt="UI-Vision Results">
-</div>
-
-### UI-I2E-Bench Results
-
-To further probe semantic reasoning, we evaluated on UI-I2E-Bench, a benchmark featuring a high proportion of implicit instructions that require reasoning beyond direct text matching. Our model's strong performance underscores AEPO's ability to handle complex, indirect commands.
-
-<div align="center">
-  <img src="https://raw.githubusercontent.com/InfiXAI/InfiGUI-G1/main/assets/results_i2e-bench.png" width="90%" alt="UI-I2E-Bench Results">
-</div>
-
-### ScreenSpot-V2 Results
-
-On the widely-used ScreenSpot-V2 benchmark, which provides comprehensive coverage across mobile, desktop, and web platforms, InfiGUI-G1 consistently outperforms strong baselines, demonstrating the broad applicability and data efficiency of our approach.
-
-<div align="center">
-  <img src="https://raw.githubusercontent.com/InfiXAI/InfiGUI-G1/main/assets/results_screenspot-v2.png" width="90%" alt="ScreenSpot-V2 Results">
+Our InfiGUI-G1 models, trained with the AEPO framework, establish new state-of-the-art results among open-source models across a diverse and challenging set of GUI grounding benchmarks:
+
+<div align="left">
+  <table style="width: 100%; max-width: 750px; border-collapse: collapse; border-top: 2px solid #212529; border-bottom: 2px solid #212529; font-family: sans-serif;">
+      <thead style="background-color: #f8f9fa;">
+          <tr style="border-bottom: 1.5px solid #212529;">
+              <th style="padding: 12px 10px; text-align: left; width: 24.9%; font-weight: 600; color: #343a40;">Model</th>
+              <th style="padding: 12px 10px; text-align: center; font-weight: 600; color: #343a40;">MMBench-GUI</th>
+              <th style="padding: 12px 10px; text-align: center; font-weight: 600; color: #343a40;">ScreenSpot-v2</th>
+              <th style="padding: 12px 10px; text-align: center; font-weight: 600; color: #343a40;">UI-Vision</th>
+              <th style="padding: 12px 10px; text-align: center; font-weight: 600; color: #343a40;">I2E-Bench</th>
+              <th style="padding: 12px 10px; text-align: center; font-weight: 600; color: #343a40;">ScreenSpot-Pro</th>
+          </tr>
+      </thead>
+      <tbody>
+          <tr>
+              <td style="padding: 10px; text-align: left;">Qwen2.5-VL-3B</td>
+              <td style="padding: 10px; text-align: center;">-</td>
+              <td style="padding: 10px; text-align: center;">80.9</td>
+              <td style="padding: 10px; text-align: center;">-</td>
+              <td style="padding: 10px; text-align: center;">41.7</td>
+              <td style="padding: 10px; text-align: center;">-</td>
+          </tr>
+          <tr>
+              <td style="padding: 10px; text-align: left;">UI-R1-E-3B</td>
+              <td style="padding: 10px; text-align: center;">-</td>
+              <td style="padding: 10px; text-align: center;">-</td>
+              <td style="padding: 10px; text-align: center;">-</td>
+              <td style="padding: 10px; text-align: center;"><u>69.1</u></td>
+              <td style="padding: 10px; text-align: center;"><u>33.5</u></td>
+          </tr>
+          <tr>
+              <td style="padding: 10px; text-align: left;">Aguvis-7B</td>
+              <td style="padding: 10px; text-align: center;"><u>45.7</u></td>
+              <td style="padding: 10px; text-align: center;">-</td>
+              <td style="padding: 10px; text-align: center;"><u>13.7</u></td>
+              <td style="padding: 10px; text-align: center;">53.2</td>
+              <td style="padding: 10px; text-align: center;">-</td>
+          </tr>
+          <tr>
+              <td style="padding: 10px; text-align: left;">OS-Atlas-7B</td>
+              <td style="padding: 10px; text-align: center;">41.4</td>
+              <td style="padding: 10px; text-align: center;"><u>85.1</u></td>
+              <td style="padding: 10px; text-align: center;">9.0</td>
+              <td style="padding: 10px; text-align: center;">58.6</td>
+              <td style="padding: 10px; text-align: center;">-</td>
+          </tr>
+          <tr>
+              <th colspan="6" style="padding: 10px 12px; text-align: left; font-style: italic; background-color: #f8f9fa; border-top: 1px solid #dee2e6; border-bottom: 1px solid #dee2e6; color: #343a40;">Ours</th>
+          </tr>
+          <tr style="background-color: #f0f8ff;">
+              <td style="padding: 10px; text-align: left;"><b>InfiGUI-G1-3B</b></td>
+              <td style="padding: 10px; text-align: center;"><b>73.4</b></td>
+              <td style="padding: 10px; text-align: center;"><b>91.1</b></td>
+              <td style="padding: 10px; text-align: center;"><b>22.0</b></td>
+              <td style="padding: 10px; text-align: center;"><b>72.6</b></td>
+              <td style="padding: 10px; text-align: center;"><b>45.2</b></td>
+          </tr>
+          <tr style="backgroundColor: #f0f8ff;">
+              <td style="padding: 10px; text-align: right;"><i>w/ Expl. Success</i></td>
+              <td style="padding: 10px; text-align: center;">81.6</td>
+              <td style="padding: 10px; text-align: center;">94.4</td>
+              <td style="padding: 10px; text-align: center;">29.7</td>
+              <td style="padding: 10px; text-align: center;">82.8</td>
+              <td style="padding: 10px; text-align: center;">52.0</td>
+          </tr>
+      </tbody>
+  </table>
 </div>
 
-## ⚙️ Evaluation
+## Evaluation
 
 This section provides instructions for reproducing the evaluation results reported in our paper.
 
@@ -307,7 +336,7 @@ python eval/eval.py \
 
 Evaluation results, including detailed logs and performance metrics, will be saved to the `./output/{model_name}/{benchmark}/` directory.
 
-## 📚 Citation Information
+## Citation Information
 
 If you find this work useful, we would be grateful if you consider citing the following papers:
 
@@ -341,6 +370,6 @@ If you find this work useful, we would be grateful if you consider citing the fo
 }
 ```
 
-## 🙏 Acknowledgements
+## Acknowledgements
 
 We would like to express our gratitude for the following open-source projects: [VERL](https://github.com/volcengine/verl), [Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL) and [vLLM](https://github.com/vllm-project/vllm).