Add model card

Signed-off-by: Gaoyang Zhang <[email protected]>

README.md

@@ -0,0 +1,121 @@
+---
+tags:
+- text-to-image
+- diffusers
+widget:
+- text: a photo of a laptop above a dog
+  output:
+    url: images/laptop-above-dog.jpg
+- text: a photo of a potted plant to the right of a motorcycle
+  output:
+    url: images/potted_plant-right-motorcycle.jpg
+- text: a photo of a sheep below a sink
+  output:
+    url: images/sheep-below-sink.jpg
+base_model: stabilityai/stable-diffusion-2-1
+license: apache-2.0
+---
+# CoMPaSS-SD2.1
+
+<Gallery />
+
+## Model description 
+
+# CoMPaSS-SD2.1
+
+\[[Project Page]\]
+\[[code]\]
+\[[arXiv]\]
+
+A UNet that enhances spatial understanding capabilities of the StableDiffusion 2.1 text-to-image
+diffusion model.  This model demonstrates significant improvements in generating images with specific
+spatial relationships between objects.
+
+## Model Details
+
+- **Base Model**: StableDiffusion 2.1
+- **Training Data**: SCOP dataset (curated from COCO)
+- **Framework**: Diffusers
+- **License**: Apache-2.0 (see [./LICENSE])
+
+## Intended Use
+
+- Generating images with accurate spatial relationships between objects
+- Creating compositions that require specific spatial arrangements
+- Enhancing the base model's spatial understanding while maintaining its other capabilities
+
+## Performance 
+
+### Key Improvements
+
+- VISOR benchmark: +105.2% relative improvement
+- T2I-CompBench Spatial: +146.2% relative improvement
+- GenEval Position: +628.6% relative improvement
+- Maintains or improves base model's image fidelity (lower FID and CMMD scores than base model)
+
+## Using the Model
+
+See our [GitHub repository][code] to get started.
+
+### Effective Prompting
+
+The model works well with:
+- Clear spatial relationship descriptors (left, right, above, below)
+- Pairs of distinct objects
+- Explicit spatial relationships (e.g., "a photo of A to the right of B")
+
+## Training Details
+
+### Training Data
+
+- Built using the SCOP (Spatial Constraints-Oriented Pairing) data engine
+- ~28,000 curated object pairs from COCO
+- Enforces criteria for:
+  - Visual significance
+  - Semantic distinction
+  - Spatial clarity
+  - Object relationships
+  - Visual balance
+
+### Training Process
+
+- Trained for 80,000 steps
+- Effective batch size of 4
+- Learning rate: 5e-6
+- Optimizer: AdamW with β₁=0.9, β₂=0.999
+- Weight decay: 1e-2
+
+## Evaluation Results
+
+| Metric | StableDiffusion 1.4 | +CoMPaSS |
+|--------|-------------|-----------|
+| VISOR uncond (⬆️) | 30.25% | **62.06%** |
+| T2I-CompBench Spatial (⬆️) | 0.13 | **0.32** |
+| GenEval Position (⬆️) | 0.07 | **0.51** |
+| FID (⬇️) | 21.65 | **16.96** |
+| CMMD (⬇️) | 0.6472 | **0.4083** |
+
+## Citation
+
+If you use this model in your research, please cite:
+```bibtex
+@article{zhang2024compass,
+  title={CoMPaSS: Enhancing Spatial Understanding in Text-to-Image Diffusion Models},
+  author={Zhang, Gaoyang and Fu, Bingtao and Fan, Qingnan and Zhang, Qi and Liu, Runxing and Gu, Hong and Zhang, Huaqi and Liu, Xinguo},
+  journal={arXiv preprint arXiv:2412.13195},
+  year={2024}
+}
+```
+
+## Contact
+
+For questions about the model, please contact <[email protected]>
+
+## Download model
+
+Weights for this model are available in Safetensors format.
+
+[./LICENSE]: <./LICENSE>
+[code]: <https://github.com/blurgyy/CoMPaSS>
+[Project page]: <https://compass.blurgy.xyz>
+[arXiv]: <https://arxiv.org/abs/2412.13195>