size_categories:n<1K format:json modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
dataset_info:
  features:
  - name: id
    dtype: string
  - name: instruction
    dtype: string
  - name: trajectory
    list:
    - name: image_id
      dtype: string
    - name: action_options
      list:
      - name: index
        dtype: int32
      - name: top_left
        sequence: int32
      - name: bottom_right
        sequence: int32
    - name: action_taken
      struct:
      - name: type
        dtype: string
      - name: value
        dtype: string
      - name: action_option_index
        dtype: int32
  splits:
  - name: train
    num_bytes: 722
    num_examples: 1
  download_size: 1432409
  dataset_size: 722
---
# BabyAGI (Dataset)

The initial demonstration dataset follows the Huggingface dataset spec, with the raw data split into two components, trajectory images and trajectory metadata. The metadata is stored in the raw dataset, and the images are stored on S3. The data is loaded using the dataloader defined in [baby_agi_dataset.py](./baby_agi_dataset.py).

**Data Layout:**

```plaintext
├── data
│   ├── metadata_0.json
│   ├── metadata_1.json
│   └── ...
├-- baby_agi_dataset.py
```

### Metadata Format (.json)

```json
[
      {
            "id": "<trajectory_id_hash>",
            "instruction": "<some instruction>",
            "trajectory": [
                {
                    "image_id": "image_id",
                    "action_options": [
                        {
                            "index": 0,
                            "top_left": [120, 340],
                            "bottom_right": [140, 440],
                        },
                        ...
                    ],
                    "action_taken": {
                        "type": "click",
                        "value": "value (only for type and scroll)",
                        "action_option_index": 0
                    }
                },
                ...
            ]
        },
]
```

## Action Types

The dataset metadata includes three types of actions: "click", "type", and "scroll". The `action_option_index` field indicates the index of the clicked element within the `action_options` list.

1. **Click**: Represents a user clicking on an element.

2. **Type**: Represents a user typing into an input field.

3. **Scroll**: Represents a user scrolling the viewport. The `value` field indicates the direction of the scroll, with "up" corresponding to a 200px scroll upwards and "down" corresponding to a 200px scroll downwards. Note that `bottom_left` and `top_right` will always be zero-arrays for these.


## Dataset Generation Pipeline

The dataset is generated through the following steps:

1. **Load Demo**: The demo is loaded from the Hugging Face dataset.
2. **Load Trace**: The trace is loaded from the Globus dataset.
3. **Process Trajectories**: For each Mind2Web (M2W) trajectory:

   a) **Map Actions**: M2W actions are mapped to Playwright trace actions using the timestamp in `dom_content.json`.

   b) **Screenshot DOM**: The DOM is "screenshoted" just before the action.

   c) **Map Candidates**: `pos_candidates` and `neg_candidates` from the M2W action metadata are mapped to HTML bounding boxes via class+id matching from the action metadata. New bounding box coordinates are obtained for each.

   d) **Craft Meta + Screenshot Pair**: The pair of metadata and screenshots is crafted and saved/appended.

4. **Save Data**: The updated data directory is saved to S3 and Hugging Face.


### Screenshots

Screenshots in this dataset are generated from the before states of Mind2Web trajectory traces. Each image has a width of 2036 and a height of 1144. For alternate screen sizes (via augmentation), padding is added to maintain the aspect ratio. This ensures that the content of the screenshot remains consistent across different screen sizes.


### Options Generation

Options in this dataset are generated from `positive_candidates` (always one) and `negative_candidates` in the Mind2Web (M2W) dataset. The M2W dataset labels *all* possible interactions on the DOM. Therefore, the 50 largest area-wise options within the viewport containing the positive candidate are selected. 

### Scrolling

The Mind2Web (M2W) dataset captures the entire DOM, so when the selected option action is not in the viewport, artificial scroll actions are created. This action has two possible values: "up" and "down". Each of which corresponds to a 200px scroll in the respective direction.


### Selecting 

The "Select" action in the Mind2Web (M2W) dataset is recorded when a user makes a selection from a dropdown list. In this dataset, we represent it as a sequence of two distinct actions in a trajectory: 

1. **Click**: The user clicks on the dropdown element.
2. **Type**: The user types the desired value followed by Enter


## Usage

To use the dataset in your Python program, you can load it using the `load_dataset` function from the `datasets` library:

```python
from datasets import load_dataset

# typically load_dataset("lukemann/baby-agi-dataset-v0"
dataset = load_dataset("lukemann/baby-agi-dataset-v0")

first_row = dataset['train'][0]
print(first_row)
```

This will load the dataset and print the first row of the training set.

For a short demo, refer to the [demo.py](./demo.py) file.