YuchengShi
/

LLaVA-v1.5-7B-Fgvc

@@ -1,27 +1,35 @@
 ---
-library_name: transformers
-tags: []
 ---
-# Fine-Grained Visual Classification on FGVC-Aircraft
-Project Page: [SelfSynthX](https://github.com/sycny/SelfSynthX).
 Paper on arXiv: [Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data](https://arxiv.org/abs/2502.14044)
-This model is a fine-tuned multimodal foundation model based on [LLaVA-1.5-7B-hf](https://huggingface.co/llava-hf/llava-1.5-7b-hf), optimized for fine-grained classification of aircraft types using the FGVC-Aircraft dataset.
-## Key Details
-- **Base Model:** LLaVA-1.5-7B
-- **Dataset:** FGVC-Aircraft (Fine-Grained Visual Classification of Aircraft)
-- **Innovation:**
-  - **Self-Synthesized Data:** Extracts and highlights distinctive aircraft-specific visual features using the Information Bottleneck principle.
-  - **Iterative Fine-Tuning:** Uses reward model-free rejection sampling to improve classification accuracy and explanation quality.
 - **Intended Use:** Identification of aircraft models with human-verifiable explanations.
-## How to Use
 ```python
 import requests
@@ -31,8 +39,8 @@ from transformers import AutoProcessor, LlavaForConditionalGeneration
 model_id = "YuchengShi/LLaVA-v1.5-7B-Fgvc"
 model = LlavaForConditionalGeneration.from_pretrained(
-    model_id,
-    torch_dtype=torch.float16,
     low_cpu_mem_usage=True,
 ).to("cuda")
 processor = AutoProcessor.from_pretrained(model_id)
@@ -55,14 +63,14 @@ output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
 print(processor.decode(output[0][2:], skip_special_tokens=True))
 ```
-## Training & Evaluation
-- **Training:** Fine-tuned using LoRA on FGVC-Aircraft with iterative rejection sampling.
 - **Evaluation:** Achieves high accuracy in distinguishing aircraft types while providing detailed, interpretable explanations.
-## Citation
-If you use this model, please cite:
 ```bibtex
 @inproceedings{

 ---
+library_name: transformers
+pipeline_tag: image-text-to-text
+license: mit
+tags:
+  - multimodal
+  - image-classification
+  - explanation
+  - visual-reasoning
+  - fine-grained-classification
+  - llava
+  - fgvc
 ---
+# Fine-Grained Visual Classification on FGVC-Aircraft
+Project Page: [SelfSynthX](https://github.com/sycny/SelfSynthX).
 Paper on arXiv: [Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data](https://arxiv.org/abs/2502.14044)
+This model is a fine-tuned multimodal foundation model based on [LLaVA-1.5-7B-hf](https://huggingface.co/llava-hf/llava-1.5-7B-hf), optimized for fine-grained classification of aircraft types using the FGVC-Aircraft dataset.
+## Key Details
+- **Base Model:** LLaVA-1.5-7B
+- **Dataset:** FGVC-Aircraft (Fine-Grained Visual Classification of Aircraft)
+- **Innovation:**
+  - **Self-Synthesized Data:** Extracts and highlights distinctive aircraft-specific visual features using the Information Bottleneck principle.
+  - **Iterative Fine-Tuning:** Uses reward model-free rejection sampling to improve classification accuracy and explanation quality.
 - **Intended Use:** Identification of aircraft models with human-verifiable explanations.
+## How to Use
 ```python
 import requests
 model_id = "YuchengShi/LLaVA-v1.5-7B-Fgvc"
 model = LlavaForConditionalGeneration.from_pretrained(
+    model_id,
+    torch_dtype=torch.float16,
     low_cpu_mem_usage=True,
 ).to("cuda")
 processor = AutoProcessor.from_pretrained(model_id)
 print(processor.decode(output[0][2:], skip_special_tokens=True))
 ```
+## Training & Evaluation
+- **Training:** Fine-tuned using LoRA on FGVC-Aircraft with iterative rejection sampling.
 - **Evaluation:** Achieves high accuracy in distinguishing aircraft types while providing detailed, interpretable explanations.
+## Citation
+If you use this model, please cite:
 ```bibtex
 @inproceedings{