license:apache-2.0 region:us
---
license: apache-2.0
---

# URLs of images containing faces from Common Crawl

![image of faces](faces.png)

## Download images

Select only first face from image and faces with min size of 40 pixels:
```python
from datasets import load_dataset

def filter_bbox(bbox, min_size=40):
    x1, x2, y1, y2 = bbox
    return x2 - x1 >= min_size and y2 - y1 >= min_size

ds = load_dataset('atom-in-the-universe/cc-faces-150k')
ds = ds.map(lambda sample: {'faces': sample['faces'][0]})
ds = ds.filter(lambda sample: filter_bbox(sample['faces']))
ds.to_parquet('cc_faces.parquet')
```

## Download using img2dataset

Install Vanga's fork of img2dataset:
```bash
pip install img2dataset git+https://github.com/vanga/img2dataset.git
``` 

Python script:
```python
from img2dataset import download
import os

output_dir = os.path.abspath("bench")

download(
    processes_count=16,
    thread_count=32,
    url_list="cc_faces.parquet",
    image_size=256,
    output_folder=output_dir,
    output_format="files",
    input_format="parquet",
    url_col="url",
    caption_col="alt",
    enable_wandb=True,
    number_sample_per_shard=1000,
    distributor="multiprocessing",
    box_col='faces
)
```