Improve model card: Add `library_name`, abstract, project page, and usage example

README.md

@@ -1,19 +1,110 @@
 ---
-license: apache-2.0
+base_model:
+- Qwen/Qwen2-VL-7B-Instruct
 datasets:
 - psp-dada/SENTINEL
 language:
 - en
-base_model:
-- Qwen/Qwen2-VL-7B-Instruct
+license: apache-2.0
 pipeline_tag: image-text-to-text
+library_name: transformers
 ---
 
-# Model Card for SENTINEL:<br> Mitigating Object Hallucinations via Sentence-Level Early Intervention <!-- omit in toc -->
+# Model Card for SENTINEL: Mitigating Object Hallucinations via Sentence-Level Early Intervention
 
 <a href='https://arxiv.org/abs/2507.12455'>
 <img src='https://img.shields.io/badge/Paper-Arxiv-purple'></a>
 <a href='https://github.com/pspdada/SENTINEL'>
 <img src='https://img.shields.io/badge/Github-Repo-Green'></a>
+<a href='https://huggingface.co/collections/psp-dada/sentinel-686ea70912079af142015286'>
+<img src='https://img.shields.io/badge/Project-HuggingFace_Collection-orange'></a>
+
+## Abstract
+
+Multimodal large language models (MLLMs) have revolutionized cross-modal understanding but continue to struggle with hallucinations - fabricated content contradicting visual inputs. Existing hallucination mitigation methods either incur prohibitive computational costs or introduce distribution mismatches between training data and model outputs. We identify a critical insight: hallucinations predominantly emerge at the early stages of text generation and propagate through subsequent outputs. To address this, we propose **SENTINEL** (**S**entence-level **E**arly i**N**tervention **T**hrough **IN**-domain pr**E**ference **L**earning), a framework that eliminates dependency on human annotations. Specifically, we first bootstrap high-quality in-domain preference pairs by iteratively sampling model outputs, validating object existence through cross-checking with two open-vocabulary detectors, and classifying sentences into hallucinated/non-hallucinated categories. Subsequently, we use context-coherent positive samples and hallucinated negative samples to build context-aware preference data iteratively. Finally, we train models using a context-aware preference loss (C-DPO) that emphasizes discriminative learning at the sentence level where hallucinations initially manifest. Experimental results show that SENTINEL can reduce hallucinations by over 90\% compared to the original model and outperforms the previous state-of-the-art method on both hallucination benchmarks and general capabilities benchmarks, demonstrating its superiority and generalization ability. The models, datasets, and code are available at this https URL .
+
+## 🚀 Overview
+
+**SENTINEL** introduces an automatic, sentence‑level early intervention strategy to prevent and mitigate object hallucinations in multimodal large language models (MLLMs). Key advantages:
+
+-   **Annotation‑free**: No human labeling required.
+-   **Model-agnostic**: Compatible with any MLLM architecture.
+-   **Efficient**: Lightweight LoRA fine‑tuning.
+
+## 🔑 Key Features
+
+-   🧠 **Early intervention halts hallucination propagation**. We find that hallucinations of MLLMs predominantly arise in early sentences and propagate through the rest of the output. SENTINEL interrupts this chain early to maximize mitigation.
+-   🔍 **In-domain contextual preference learning without human labels**. SENTINEL constructs hallucinated/factual samples via detector cross-validation and builds context-aware preference data without relying on proprietary LLMs or manual annotations.
+-   💡 **Context matters: rich coherence drives robustness**. By prioritizing context-coherent positive samples over hallucinated ones, SENTINEL significantly boosts generalization.
+-   ♻️ **Iterative contextual bootstrapping for diverse hallucination-free contexts**. Our pipeline dynamically grows non-hallucinated contexts and expands coverage across varied scenes, improving robustness across generations.
+-   📊 **State-of-the-art results across benchmarks**. SENTINEL achieves **up to 92% reduction** in hallucinations and outperforms prior SOTA methods across Object HalBench, AMBER, and HallusionBench, while maintaining or improving general task performance.
+
+## 📦 Model Weights
+
+This model is a LoRA adapter for `Qwen/Qwen2-VL-7B-Instruct`. It can be seamlessly plugged into the corresponding base model for inference or further fine-tuning.
+
+## Usage
+
+You can use this model with the Hugging Face `transformers` library. Since this is a LoRA adapter for a Qwen2-VL model, ensure `trust_remote_code=True` is used for proper loading.
+
+```python
+from transformers import AutoModelForCausalLM, AutoProcessor
+from PIL import Image
+import requests
+from io import BytesIO
+
+# Load model and processor.
+# For Qwen2-VL models, trust_remote_code=True is required due to custom code.
+model_id = "psp-dada/Qwen2-VL-7B-SENTINEL"
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    torch_dtype="auto",
+    device_map="auto",
+    trust_remote_code=True
+)
+processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
+
+# Example image from the SENTINEL GitHub repository
+image_url = "https://raw.githubusercontent.com/pspdada/SENTINEL/main/docs/figures/figure1.png"
+response = requests.get(image_url)
+image = Image.open(BytesIO(response.content)).convert("RGB")
+
+# Prepare messages following the Qwen-VL chat template
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "image", "image": image},
+            {"type": "text", "text": "Describe this image in detail."}
+        ],
+    }
+]
+
+# Apply chat template and process inputs
+text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+inputs = processor(text=[text], images=[image], padding=True, return_tensors="pt")
+inputs = inputs.to(model.device)
+
+# Generate response
+generated_ids = model.generate(**inputs, max_new_tokens=256, do_sample=False)
+
+# Decode and print the output
+# The generated_ids contain the input_ids as prefix. Trim them for clean output.
+output_text = processor.batch_decode(
+    generated_ids[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True, clean_up_tokenization_spaces=False
+)[0]
+print(output_text)
+```
+
+## 📝 Citation
+
+If you find our model/code/data/paper helpful, please consider cite our papers 📝 and star us ⭐️！
 
-For the details of this model, please refer to the [documentation](https://github.com/pspdada/SENTINEL?tab=readme-ov-file#-model-weights) of the GitHub repo.
+```bibtex
+@article{peng2025mitigating,
+  title={Mitigating Object Hallucinations via Sentence-Level Early Intervention},
+  author={Peng, Shangpin and Yang, Senqiao and Jiang, Li and Tian, Zhuotao},
+  journal={arXiv preprint arXiv:2507.12455},
+  year={2025}
+}
+```