README.md

# Concept Decomposition for Visual Exploration and Inspiration

<a href="https://inspirationtree.github.io/inspirationtree/"><img src="https://img.shields.io/static/v1?label=Project&message=Website&color=blue"></a>
<a href="https://arxiv.org/abs/2305.18203"><img src="https://img.shields.io/badge/arXiv-2305.16311-b31b1b.svg"></a>
<a href="https://www.apache.org/licenses/LICENSE-2.0.txt"><img src="https://img.shields.io/badge/License-Apache-yellow"></a>
<!-- Official implementation. -->
<br>


<p align="center">
<img src="repo_images/teaser.jpeg" width="90%"/>  
  
> <a href="https://inspirationtree.github.io/inspirationtree/">**Concept Decomposition for Visual Exploration and Inspiration**</a>
>
> <a href="https://yael-vinker.github.io/website/">Yael Vinker</a>,
<a href="https://scholar.google.com/citations?user=imBjSgUAAAAJ&hl=ru">Andrey Voynov</a>,
<a href="https://danielcohenor.com/">Daniel Cohen-Or</a>,
<a href="https://faculty.runi.ac.il/arik/site/index.asp">Ariel Shamir</a>
> <br>
> Our method provides a tree-structured visual exploration space for a given unique concept. The nodes of the tree ("v_i") are newly learned textual vector embeddings, injected to the latent space of a pretrained text-to-image model. The nodes encode different aspects of the subject of interest. Through examining combinations within > and across trees, the different aspects can inspire the creation of new designs and concepts, as can be seen below.
</p>

# Setup
```
git clone https://github.com/yael-vinker/inspiration_tree.git
```

## Environment (with pip)
Our code relies on the enviornment in the official [Stable Diffusion repository](https://github.com/CompVis/stable-diffusion). To set up their environment, please run:
```
python -m venv .tree_venv
source .tree_venv/bin/activate
pip install -r requirements.txt
```
Technical details:
* CUDA 11.6
* torch 1.7.1+cu110

**Hugging Face Diffusers Library**  
Our code relies on the [diffusers](https://github.com/huggingface/diffusers) library and the official [Stable Diffusion v1.4](https://huggingface.co/CompVis/stable-diffusion-v1-4) model.

# Usage
<p align="center">
<img src="repo_images/usage.jpg" width="50%"/>  
<br>
This code will allow you to generate a tree per concept, and play with the different prompts and combinations for the generated tree (under "inspiration_tree_playground.ipynb").
</p>


## Pretrained Models and Datasets
As part of our code release and to assist with comparisons, we have also provided some of the trained models and datasets used in the paper. 

All of our models (learned tokens from the paper) can be found undeer the "learned_tokens" directory. The notebook inspiration_tree_playground.ipynb shows how to load them and reproduce the results from the paper (under "Play with learned aspects from the paper"). 
All datasets used from Textual Inversion can be found under "datasets".

## Generate your tree
The logic for generating the tree is under "main_multiseed.py", which runs the framwork for a <b>single node</b>.
You can generate the tree by passing your own parameters to main_multiseed.py. An example is given in "run_decompose.sh":
```
python main_multiseed.py --parent_data_dir "cat_sculpture/" --node v0 --test_name "v0" --GPU_ID "${GPU_ID}" --multiprocess 0
```
Notes:
- The "test_name" should be identical to the chosen node
Results:
- The results will be saved to "outputs/<parent_data_dir>/<node>"
- In "final_samples.jpg" you can see a batch of random samples of the learned nodes
- Under "consistency_test" we save the results of the seed selection procedure
- Once training is finished, you can continue with splitting the generated nodes as well

## Inference
We made a notebook ("inspiration_tree_playground.ipynb") to play with the results, and this should be the most convenient option.
Notes: 
- The notebook shows the concepts learned in each node
- You can also load checkpoints from early iterations
- You can generate new concepts using natural language sentences

# Acknowledgements
Our code builds on the [diffusers implementation of textual inversion](https://github.com/huggingface/diffusers/tree/main/examples/textual_inversion) 

## Citation
If you find this useful for your research, please cite the following:
```bibtex
@article{vinker2023concept,
  title={Concept Decomposition for Visual Exploration and Inspiration},
  author={Yael Vinker and Andrey Voynov and Daniel Cohen-Or and Ariel Shamir},
  journal={arXiv preprint arXiv:2305.18203},
  year={2023}
}
```

## Disclaimer

This is not an officially supported Google product.