size_categories:1K<n<10K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us
---
dataset_info:
  features:
  - name: source
    dtype: string
  - name: messages
    list:
    - name: content
      dtype: string
    - name: role
      dtype: string
  - name: conversations
    list:
    - name: from
      dtype: string
    - name: value
      dtype: string
  splits:
  - name: train
    num_bytes: 35640524
    num_examples: 4854
  download_size: 16804823
  dataset_size: 35640524
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
---
# Small Chat

This is a small multi-turn chat dataset based on publicly available data sources. It contains approximately 5,000 examples, allowing for quick training and evaluation of base models to assess their chat capabilities.

The dataset prioritizes *quality* over *quantity*.

It was compiled using the following sources:

- HuggingFaceH4/OpenHermes-2.5-1k-longest
- LDJnr/Pure-Dove

I performed some preprocessing to merge the datasets into a consistent format and to remove artifacts from ShareGPT.

This is an evolving dataset, and I am open to incorporating other high-quality datasets. If you find any other high-quality datasets that could be a good fit, please contact me.

## Dataset Structure

The data has the same structure as HuggingFaceH4/OpenHermes-2.5-1k-longest (ShareGPT format)
Each example has the following structure:

```json
{
    "source": "Dove_final",
    "messages": [
        {
            "role": "user",
            "content": "If I have a list of dicts in python with a field \"id\" and list \"tags\", how can I add a new field to each dict \"duplicates\" which is a list of all ids which have a duplicate tag to this dict?"
        },
        {
            "role": "assistant",
            "content": "You can achieve this by iterating through the list of dictionaries and comparing the tags of each dictionary with the others. Here's how you can do it:\n\n```python\nlist_of_dicts = [\n    {\"id\": 1, \"tags\": [\"a\", \"b\"]},\n    {\"id\": 2, \"tags\": [\"b\", \"c\"]},\n    {\"id\": 3, \"tags\": [\"c\", \"d\"]},\n    {\"id\": 4, \"tags\": [\"a\", \"e\"]}\n]\n\nfor i, current_dict in enumerate(list_of_dicts):\n    current_dict[\"duplicates\"] = []\n    for other_dict in list_of_dicts[:i] + list_of_dicts[i+1:]:\n        if set(current_dict[\"tags\"]).intersection(set(other_dict[\"tags\"])):\n            current_dict[\"duplicates\"].append(other_dict[\"id\"])\n\nprint(list_of_dicts)\n```\n\nThis code will modify the `list_of_dicts` to include the \"duplicates\" field for each dictionary:\n\n```python\n[\n    {\"id\": 1, \"tags\": [\"a\", \"b\"], \"duplicates\": [2, 4]},\n    {\"id\": 2, \"tags\": [\"b\", \"c\"], \"duplicates\": [1, 3]},\n    {\"id\": 3, \"tags\": [\"c\", \"d\"], \"duplicates\": [2]},\n    {\"id\": 4, \"tags\": [\"a\", \"e\"], \"duplicates\": [1]}\n]\n```\n\nThis result shows that dictionaries with ids 1, 2, and 4 have duplicate tags with other dictionaries (ids 2, 1, and 1 respectively)."
        }
    ]
}
```

Here, `source` and `category` refer to metadata where the datapoint came from, while `messages` consists of multi-turn conversations that can be wrapped in a chat template like ChatML for training.
There are currently no system messages in the training set.