task_categories:image-to-image task_categories:text-to-image language:en license:cc-by-4.0 size_categories:100K<n<1M arxiv:2312.03849 region:us
---
license: cc-by-4.0
task_categories:
- image-to-image
- text-to-image
language:
- en
size_categories:
- 100K<n<1M
viewer: false
---

# LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning

### ECCV 2024 (Oral, Best Paper Finalist)

[Project Page](https://bolinlai.github.io/Lego_EgoActGen/) | [Paper](https://arxiv.org/pdf/2312.03849) | [Code](https://github.com/BolinLai/LEGO/tree/main)

[Bolin Lai](https://bolinlai.github.io/), [Xiaoliang Dai](https://sites.google.com/view/xiaoliangdai/), [Lawrence Chen](https://www.lawrencechen.me/), [Guan Pang](https://scholar.google.com/citations?user=7v1LZxUAAAAJ&hl=en), [James M. Rehg](https://rehg.org/), [Miao Liu](https://aptx4869lm.github.io/)

This repo is the dataset for egocentric action frame generation. This dataset consists of ~145k pairs of egocentric video frames (start frame + action frame) and action labels sourced from [Ego4D](https://ego4d-data.org/) and [Epic-Kitchens](https://epic-kitchens.github.io/2025). We also provide the detailed action descriptions generated by GPT-3.5 or our instruction-tuned LLM.

Please refer to our [paper](https://arxiv.org/pdf/2312.03849) for more details, and our code on [github](https://github.com/BolinLai/LEGO) for instructions on how to use it. More repos are available in this [collection](https://huggingface.co/collections/bolinlai/lego-67b386cf642909c56776f754).

If you find LEGO useful for your work, please cite using this BibTeX.

```BibTex
@inproceedings{lai2024lego,
  title={Lego: Learning egocentric action frame generation via visual instruction tuning},
  author={Lai, Bolin and Dai, Xiaoliang and Chen, Lawrence and Pang, Guan and Rehg, James M and Liu, Miao},
  booktitle={European Conference on Computer Vision},
  pages={135--155},
  year={2024},
  organization={Springer}
}
```

The bibtex for the original dataset papers:

```BibTex
@inproceedings{grauman2022ego4d,
  title={Ego4d: Around the world in 3,000 hours of egocentric video},
  author={Grauman, Kristen and Westbury, Andrew and Byrne, Eugene and Chavis, Zachary and Furnari, Antonino and Girdhar, Rohit and Hamburger, Jackson and Jiang, Hao and Liu, Miao and Liu, Xingyu and others},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={18995--19012},
  year={2022}
}
```
```BibTex
@inproceedings{damen2018scaling,
  title={Scaling egocentric vision: The epic-kitchens dataset},
  author={Damen, Dima and Doughty, Hazel and Farinella, Giovanni Maria and Fidler, Sanja and Furnari, Antonino and Kazakos, Evangelos and Moltisanti, Davide and Munro, Jonathan and Perrett, Toby and Price, Will and others},
  booktitle={Proceedings of the European conference on computer vision (ECCV)},
  pages={720--736},
  year={2018}
}
```