language:en license:apache-2.0 size_categories:n<1K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
dataset_info:
  features:
  - name: name
    dtype: string
  - name: url
    dtype: string
  - name: url_full
    dtype: string
  - name: llms_txt
    dtype: string
  - name: llms_txt_parsed
    dtype: string
  - name: llms_txt_full
    dtype: string
  splits:
  - name: train
    num_bytes: 53928865
    num_examples: 91
  download_size: 19623460
  dataset_size: 53928865
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
license: apache-2.0
language:
- en
pretty_name: llms.txt
size_categories:
- n<1K
---

# Dataset with /llms.txt

This dataset contains the *llms.txt* and *llms-full.txt* files obtained from [directory.llmstxt.cloud](https://directory.llmstxt.cloud/).

The `llms.txt` defined at [https://llmstxt.org/](https://llmstxt.org/):

> A proposal to standardise on using an /llms.txt file to provide information to help LLMs use a website at inference time.

Some sites are starting to adopt this standard, and they are being tracked by [directory.llmstxt.cloud](https://directory.llmstxt.cloud/),
so we can scrape and collect them in a dataset.

## How it was obtained

By running [dataset.py](https://huggingface.co/datasets/plaguss/llms-txt/blob/main/dataset.py) that can be found in this repo:

```bash
uv run dataset.py
```

1. It scrapes [directory.llmstxt.cloud](https://directory.llmstxt.cloud/) and extracts the urls of the tracked sites.

2. The *llms.txt* and *llms-full.txt* found from each site are downloaded.

3. The *llms.txt* files are parsed using [llms-txt-rs](https://github.com/plaguss/llms-txt-rs).

4. The parsed content is encoded as a json string to avoid mixing fields.

5. Save records as a dataset and push it to the hub.

## How to use it

Keep in mind, the `llms_txt_parsed` field must be transformed a dict:

```python
import json
from datasets import load_dataset

ds = load_dataset("plaguss/llms-txt", split="train")

def to_dict(row):
    row["llms_txt_parsed"] = json.loads(row["llms_txt_parsed"])
    return row

ds = ds.map(to_dict)
```

### Example row

```json
{
  "name": "llms.txt",
  "url": "https://llmstxt.org/llms.txt",
  "url_full": null,
  "llms_txt": "# llms.txt\n\n> A proposal that those interested in providing LLM-friendly content add a /llms.txt file to their site. This is a markdown file that provides brief background information and guidance, along with links to markdown files providing more detailed information.\n\n## Docs\n\n- [llms.txt proposal](https://llmstxt.org/index.md): The proposal for llms.txt\n- [Python library docs](https://llmstxt.org/intro.html.md): Docs for `llms-txt` python lib\n- [ed demo](https://llmstxt.org/ed-commonmark.md): Tongue-in-cheek example of how llms.txt could be used in the classic `ed` editor, used to show how editors could incorporate llms.txt in general.\n\n",
  "llms_txt_parsed": {
    "title": "llms.txt",
    "summary": "A proposal that those interested in providing LLM-friendly content add a /llms.txt file to their site. This is a markdown file that provides brief background information and guidance, along with links to markdown files providing more detailed information.",
    "info": null,
    "sections": {
      "Docs": [
        {
          "title": "llms.txt proposal",
          "url": "https://llmstxt.org/index.md",
          "desc": "The proposal for llms.txt"
        },
        {
          "title": "Python library docs",
          "url": "https://llmstxt.org/intro.html.md",
          "desc": "Docs for `llms-txt` python lib"
        },
        {
          "title": "ed demo",
          "url": "https://llmstxt.org/ed-commonmark.md",
          "desc": "Tongue-in-cheek example of how llms.txt could be used in the classic `ed` editor, used to show how editors could incorporate llms.txt in general."
        }
      ]
    }
  },
  "llms_txt_full": null
}
```
