language:en license:odc-by region:us
---
license: odc-by
language:
- en
---

# DuckDB datasets for `id` querying on FineWeb

This repo contains DuckDB databases to check whether a given WARC UID exists in a FineWeb dump. Usage example
is given below, but note especially that if you are using URNs (likely, if you are working with CommonCrawl data),
then you first have to extract the UID (the `id` column is of type `UUID` in the databases).


## Download

All files:

```shell
huggingface-cli download BramVanroy/fineweb-duckdbs --local-dir duckdbs/fineweb/ --include *.duckdb --repo-type dataset
```

One file:

```shell
huggingface-cli download BramVanroy/fineweb-duckdbs fw-CC-MAIN-2024-51.duckdb --local-dir duckdbs/fineweb/ --repo-type dataset
```


## Usage

Originally developed to be used with a library to look for Creative Commons licensing information: https://github.com/BramVanroy/CommonCrawl-CreativeCommons/

```python
import re
import duckdb


uid_re = re.compile(r"<urn:uuid:([a-zA-Z0-9]{8}-?[a-zA-Z0-9]{4}-?[a-zA-Z0-9]{4}-?[a-zA-Z0-9]{4}-?[a-zA-Z0-9]{12})>")

duckdb_path = "duckdbs/fineweb/fw-CC-MAIN-2019-30.duckdb"
con = duckdb.connect(duckdb_path, read_only=True)

uuid_urn = "<urn:uuid:93081b76-bc46-4d2f-98f0-2d9687b80967>"
# !! Important: extract the UUID from the URN
uid = uid_re.sub("\\1", uuid_urn).replace("-", "")

query = "SELECT EXISTS (SELECT 1 FROM data WHERE id = ?)"
exists = bool(con.execute(query, uid).fetchone()[0])

print(f"Does ID {uid} exist? {exists}")
# Does ID 93081b76bc464d2f98f0-2d9687b80967 exist? True
```