task_categories:text-generation language:en license:mit size_categories:10K<n<100K modality:text arxiv:2501.00522 region:us
---
license: mit
task_categories:
- text-generation
language:
- en
pretty_name: TinyHelen
size_categories:
- 10K<n<100K
---

# TinyHelen

This repository contains the data for the paper [TinyHelen's First Curriculum: Training and Evaluating Tiny Language Models in a Simpler Language Environment](https://arxiv.org/abs/2501.00522).

## Overview

TinyHelen aims to train and evaluate small language models using a simplified text dataset that mimics how children learn language. The dataset reduces linguistic complexity, improving training efficiency and reducing model and data size requirements.

## Key Contributions

- **Simplified Data Pipeline**: A "no noise, low complexity" approach simplifies training data, improving learning efficiency.
- **TinyHelen Dataset Suite**: Includes:
  - 71M dataset for pre-training
  - 7M dataset for instruction-tuning
  - Benchmarks for linguistic proficiency and instruction-following ability
- **Instruction-Following Models**: Initial models show progress toward self-evolving agents.
- **Low-Resource Testing**: The dataset enables testing model architectures and curriculum learning strategies with reduced data and computation.

## Citation

If you use this dataset, please cite our paper:

```
@article{yang2024tinyhelen,
  title={TinyHelen's First Curriculum: Training and Evaluating Tiny Language Models in a Simpler Language Environment},
  author={Yang, Ke and Kindratenko, Volodymyr and Zhai, ChengXiang},
  journal={arXiv preprint arXiv:2501.00522},
  year={2024}
}
```