size_categories:1K<n<10K format:parquet modality:text library:datasets library:pandas library:mlcroissant library:polars arxiv:2502.12143 region:us
---
dataset_info:
  features:
  - name: problem
    dtype: string
  - name: solution
    dtype: string
  splits:
  - name: train
    num_bytes: 22356471
    num_examples: 5383
  download_size: 9974639
  dataset_size: 22356471
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
---

# Small Model Learnability Gap

**Arxiv Technical Report:** [https://arxiv.org/pdf/2502.12143v1](https://arxiv.org/pdf/2502.12143v1)

<details>
  <summary><strong>Abstract (Click Here)</strong></summary>
  
Large language models (LLMs) excel in complex reasoning tasks, and distilling their reasoning capabilities into smaller models has shown promise. However, we uncover an interesting phenomenon, which we term the **Small Model Learnability Gap**: small models (≤3B parameters) do not consistently benefit from long chain-of-thought (CoT) reasoning or distillation from larger models. Instead, they perform better when fine-tuned on shorter, simpler reasoning chains that better align with their intrinsic learning capacity. To address this, we propose **Mix Distillation**, a simple yet effective strategy that balances reasoning complexity by combining long and short CoT examples or reasoning from both larger and smaller models. Our experiments demonstrate that Mix Distillation significantly improves small model reasoning performance compared to training on either data alone. These findings highlight the limitations of direct strong model distillation and underscore the importance of adapting reasoning complexity for effective reasoning capability transfer.
  
</details>

## Dataset Details


We utilize the problems in the **7.5k training split** from the MATH dataset and use **Qwen/QwQ-32B-Preview** to generate long chain-of-thought (CoT) answers via reject sampling. For full implementation details, please refer to our paper.

The prompt used to instruct the teacher model is:

> Solve the following math problem. Present the final answer in the format: Final Answer: \boxed{your_answer}.\nProblem: {{problem}}\nAnswer:

The motivation for developing this dataset is to investigate the effects of long versus short CoT data on different student models. This dataset uses the same math problems as the corresponding short CoT dataset and we ensure all the answers from both two dataset are correct.

You can find the model SFT checkpoint fine-tuned using this dataset [here](https://huggingface.co/collections/UWNSL/small-model-learnability-gap-models-67b8b7fb21956199df8333c7).

## Task Category
**Math**

## Language
**English**

## License

Please follow the **Tongyi Qianwen License Agreement** and **CC BY-NC 4.0**.

## 📚 Citation

If you find the model or data useful, please cite our paper:

```bibtex
@misc{li2025smallmodelsstrugglelearn,
      title={Small Models Struggle to Learn from Strong Reasoners},
      author={Yuetai Li and Xiang Yue and Zhangchen Xu and Fengqing Jiang and Luyao Niu and Bill Yuchen Lin and Bhaskar Ramasubramanian and Radha Poovendran},
      year={2025},
      eprint={2502.12143},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2502.12143},
}