language:en license:unlicense size_categories:1M<n<10M format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars region:us synthetic rlaif
---
size_categories: n<1K
dataset_info:
  features:
  - name: prompt
    dtype: string
  - name: chosen
    dtype: string
  - name: rejected
    dtype: string
  splits:
  - name: train
    num_bytes: 229727520
    num_examples: 1166807
  - name: test
    num_bytes: 39666
    num_examples: 200
  download_size: 37937962
  dataset_size: 229767186
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  - split: test
    path: data/test-*
tags:
- synthetic
- rlaif
license: unlicense
language:
- en
---

# Dataset Card for count_letters_in_word

## Dataset Summary

This dataset was generated using the nltk words corpus and a custom script to create tasks that require counting letters in words. Each example prompts the language model to count occurrences of specific letters within words. For added complexity, in 10% of cases, a letter that does not appear in the word is included. 

The dataset is structured to provide both `chosen` (correct counts) and `rejected` (incorrect counts) completions. Errors in the `rejected` completions are randomly selected within the range of {-2, -1, 1, 2} and adjusted to ensure that the counts remain valid (i.e., no negative values or counts that exceed the word length).

The primary goal of this dataset is to fine-tune language models for tasks that reinforce basic cognitive skills. By focusing on tasks typically learned in early childhood, such as letter counting, we aim to improve models' overall reasoning and language comprehension capabilities.

## Dataset structure

The dataset contains the following features:
- `prompt` (string): The question posed to the language model.
- `chosen` (string): The correct completion or answer.
- `rejected` (string): An incorrect completion or answer.

### Example

Each example is structured as follows:

```json
{
    "prompt": "Answer the following question: How many times does the letter 'i' appear in the word 'suboptimal'?",
    "chosen": "The letter 'i' appears 1 times in the word 'suboptimal'.",
    "rejected": "The letter 'i' appears 2 times in the word 'suboptimal'."
}
```

### Splits

The dataset includes two splits:
- `train`: 1,166,803 examples
- `test`: 200 examples

## Usage

You can load the dataset using the `datasets` library as follows:

```python
from datasets import load_dataset

ds = load_dataset("JeanIbarz/count_letters_in_word", "default")
```

Or simply:

```python
from datasets import load_dataset

ds = load_dataset("JeanIbarz/count_letters_in_word")
```

## Motivation

This dataset is part of ongoing efforts to enhance language model capabilities through preference optimization. Initial experiments using Supervised Fine-Tuning (SFT) on smaller datasets (e.g., 1,000 examples) and the gpt-4o-mini model showed limited performance improvements. We believe that preference-based fine-tuning, leveraging a larger dataset, can provide stronger signals for learning this specific counting task, leading to more effective model training and better generalization.