language:en license:cc-by-sa-4.0 size_categories:100K<n<1M arxiv:2306.03514 arxiv:2303.05499 arxiv:2401.10891 arxiv:2312.02010 region:us
---
license: cc-by-sa-4.0
language:
- en
size_categories:
- 100K<n<1M
configs:
  - config_name: instructions
    data_files:
      - split: open_ended_video_instruction
        path: p1_train_reformat.json
      - split: navigable_action_instruction
        path: p1_train_colmap_trajectory_tour3d.json
---


# RoomTour3D: Geometry-Aware Video-Instruction Tuning for Embodied Navigation

![image/png](https://cdn-uploads.huggingface.co/production/uploads/67402ae297acbbb4c4118c54/PLR4g2Nu98zQcNpducdNb.png)

## Dataset Description

This is only for anonymous review.

We are excited to release a new video-instruction dataset for embodied navigation. 
It contains our ongoing efforts in collecting and annotating room tour videos for training an open-world navigation agent. 
This release contains our generated geometry-aware video-instruction data from 1847 room tour scenes.
We also release all the intermediate products, such as 3D scene reconstruction using COLMAP, relative deption estimation, open-world object tags and localization.

## Annotation Format <a name="annotation-format"></a>
The released data is organized in the following structure
- **colmap_reconstruction** each separated zip file contains the corresponding reconstructed 3D scenes for the video.
- **ram_grounding_p1** each separated pickle file contains the per-frame (3 fps) object tags using [RAM](https://arxiv.org/abs/2306.03514) and the grounded object bounding boxes using [Grounding-DINO](https://arxiv.org/abs/2303.05499) for the video.
- **relative_depth_estimation** each separated pickle file contains the per-frame (3 fps) relative depth estimation using [depth-anything-model](https://arxiv.org/abs/2401.10891) for the video.
- **trajectories** contains the collated trajectories and instructions per video in the format of [NaviLLM](https://arxiv.org/abs/2312.02010) training.

### colmap_reconstruction
We show an example file structure below, which follows the original COLMAP output structure to ease visualization and processing.

```bash
- -rrFiQpQmr0_0_100
  - sparse # reconstructed sparse models
    - 0 # original reconstructed sub-model 0
    - 1+2 # merged models from sub-model 1 and sub-model 2
- -rrFiQpQmr0_90_190
- -rrFiQpQmr0_180_280
...
- -rrFiQpQmr0_360_460%450_522 # merged models from video clip 360_460 and 450_522
  - sparse 
    - 0%0 # merged model from 360_460-0 and 450_522-0
```

### ram_grounding_p1
We show the structure of the pickle file for each video, as `{'frame_name': [boxes, detction_confidence, tags]}`

```json
{
  ...,
  "output_frame_0514.png":[tensor([[0.6810, 0.4731, 0.1097, 0.1149],
                            ...
                          [0.5800, 0.8618, 0.1217, 0.2756]]), tensor([0.5931, ..., 0.2586]),
                          ['microwave(0.59)', ..., 'stool(0.25)']
                          ],
  ...
}
```


### relative_depth_estimation
We show the structure of the pickle file for each video, as

```json
{
  ...,
  "output_frame_0514.png": PIL.Image,
  ...
}

```


### trajectories
We show the structure of the collated trajectory JSON file, as

- Open-ended Video-Instruction data: p1_train_reformat.json

```json
[
  ...,
  {'annotation': [{'answers': ['Move forward from an initial position near a man, veering slightly to the right past a car and multiple flower beds, approaching a building and an entrance in the distance, while passing by various plants and following a path pavement, and finally nearing a house exterior and a stone before arriving at a doorway entrance with the building still in the far distance.'],
                   'question': 'Describe the camera movement by listing the objects that disappear from view as it pans in one direction.',
                   'question_id': 'train--0Z00G94Nl8-0'}],
   'image_info': [{'image_id': 'output_frame_0001.png'},
                  {'image_id': 'output_frame_0201.png'},
                  {'image_id': 'output_frame_0207.png'},
                  {'image_id': 'output_frame_0213.png'},
                  {'image_id': 'output_frame_0219.png'},
                  {'image_id': 'output_frame_0225.png'}],
   'seqence_id': '-0Z00G94Nl8_000',
   'type': 'video_desc',
   'videoId': '-0Z00G94Nl8'}
...
]

```

- Navigable Action-Instruction data: p1_train_colmap_trajectory_tour3d.json

```json
[
  ...,
  {"path": # frames selected by geometry-aware approach
          ["-0Z00G94Nl8_output_frame_0373.png",
           "-0Z00G94Nl8_output_frame_0378.png",
           "-0Z00G94Nl8_output_frame_0407.png",
           "-0Z00G94Nl8_output_frame_0416.png",
           "-0Z00G94Nl8_output_frame_0430.png",
           "-0Z00G94Nl8_output_frame_0442.png"],
   "videoId": "-0Z00G94Nl8",
   "path_id": "-0Z00G94Nl8_003",
   "instructions": ["Navigate through the hallway, initially observing a clock in the distance, then passing a doorway to the right and approaching a staircase ahead. Continue past an armchair and under a chandelier, moving towards an archway and a balustrade, before reaching a more confined space with a cabinet and a closer doorway. Enter the bedroom, where you encounter a dresser and a vase, with a mirror to the right and a window with curtains ahead. Move deeper into the bedroom, drawing nearer to a bed adorned with pillows and a chandelier overhead, while a window with curtains is now to the right. Finally, align with the right side of the bedroom, where the bed, lamp, nightstand, and art are in the distance, and a closet comes"],
   "longId": "-0Z00G94Nl8_90_190-0|389%393%407", # id of the navigable step, consisting of colmap model id (-0Z00G94Nl8_90_190-0) and navigable actions (389,393, 407)
   "heading": 0.0,
   "optView": "-0Z00G94Nl8_output_frame_0407.png" # optimizing target, the frame id (407)
  }
...
]
```

### geometry 

We show the structure of the collated geometry information of each frame, which would be indexed via navigable action-instruction data `path`, as
```json
[
    ...,
    # indexed by colmap model id, {video_clip_id}-{sparse_model_id}, see https://huggingface.co/datasets/roomtour3d/roomtour3d#colmap_reconstruction
    '-0Z00G94Nl8_0_100-0': {
          ...,
          # indexed by frame id in integer
          220:{'real_world_position':[0.798788158796955, 0.546947373343179, -7.9179695594989665], # projected camera location in meters, comparable to cameras in its specific colmap model, i.e., 
               'pos': [0.798788158796955, 0.546947373343179, -7.9179695594989665], # same to real_world_position
               'camera_world_position': array([ 0.4411149 ,  0.30204082, -4.37254142]),
               'yaw': 0.29741764247264585, # transformed into euc
               'pitch': -1.3533569345478835
              }
          ...,
     },
    ...
]
```


---

## Videos Downloading <a name="videos-downloading"></a>

You can download it on your own using the YouTube video id list [here](https://huggingface.co/datasets/anonymous-submission-usage/RoomTour3D/blob/main/p1_video_ids.csv).


---

## Opt-Out Approach <a name="opt-out-approach"></a>

We will provide an opt-out-approach, in case someone featured in any of our video annotations or holding copyright to a video, wishes to have its annotation removed from our dataset.

---

## License <a name="license"></a>

Our text annotations are licensed under a [Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) License](https://creativecommons.org/licenses/by-nc-sa/4.0/). They are available strictly for non-commercial research.

---
