Update README.md

README.md

@@ -1,7 +1,7 @@
 ---
 license: apache-2.0
 datasets:
-- open-r1/reasoning-mix
+- open-r1/Mixture-of-Reasons
 language:
 - en
 base_model:
@@ -11,13 +11,15 @@ library_name: transformers
 
 <img src="open-r1-thumbnail.png" alt="Centered Image" style="display: block; margin: 0 auto;" width="300">
 
-# OpenR1-Distill-7B
+# Model summary
 
-OpenR1-Distill-7B is a post-trained version of [Qwen/Qwen2.5-Math-7B](https://huggingface.co/Qwen/Qwen2.5-Math-7B) on [Mixture-of-Reasons](https://huggingface.co/datasets/open-r1/Mixture-of-Reasons); a curated dataset of 350k reasoning traces distilled from [DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1) in the domains of mathematics, coding, and science. This model matches or exceeds the performance of [DeepSeek's 7B distilled model](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B)
+OpenR1-Distill-7B is post-trained version of [Qwen/Qwen2.5-Math-7B](https://huggingface.co/Qwen/Qwen2.5-Math-7B) on [Mixture-of-Thoughts](https://huggingface.co/datasets/open-r1/Mixture-of-Thoughts): a curated dataset of 350k verified reasoning traces distilled from [DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1). The dataset spans tasks in mathematics, coding, and science, and is designed to teach language models to reason step-by-step.
+
+OpenR1-Distill-7B replicates the reasoning capabilities of [deepseek-ai/DeepSeek-R1-Distill-Qwen-7B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B) while remaining fully open and reproducible. It is ideal for research on inference-time compute and reinforcement learning with verifiable rewards (RLVR).
 
 ## Model description
 
-- **Model type:** A 7B parameter GPT-like model fine-tuned on a mix of publicly available, synthetic datasets.
+- **Model type:** A 7B parameter GPT-like model, post-trained on a mix of publicly available, synthetic datasets.
 - **Language(s) (NLP):** Primarily English
 - **License:** Apache 2.0
 - **Finetuned from model:** a [variant](https://huggingface.co/open-r1/Qwen2.5-Math-7B-RoPE-300k) of [Qwen/Qwen2.5-Math-7B](https://huggingface.co/Qwen/Qwen2.5-Math-7B), whose RoPE base frequency was extended to 300k to enable training on a context of 32k tokens.
@@ -26,99 +28,125 @@ OpenR1-Distill-7B is a post-trained version of [Qwen/Qwen2.5-Math-7B](https://hu
 
 <!-- Provide the basic links for the model. -->
 
-- **Repository:** https://github.com/huggingface/alignment-handbook
-- **Demo:** https://huggingface.co/spaces/HuggingFaceH4/zephyr-chat
-- **Chatbot Arena:** Evaluate Zephyr 7B against 10+ LLMs in the LMSYS arena: http://arena.lmsys.org
-
-## Performance
-
-At the time of release, Zephyr-7B-β is the highest ranked 7B chat model on the [MT-Bench](https://huggingface.co/spaces/lmsys/mt-bench) and [AlpacaEval](https://tatsu-lab.github.io/alpaca_eval/) benchmarks:
-
-| Model | AIME 2024 | MATH-500 | GPQA-D | LiveCodeBench |
-| :---- | :----: | :----: | :----: | :----: |
-| OpenR1-Distill-7B | 52.66 | 89 | 52.78 | X |
-| DeepSeek-R1-Distill-Qwen-7B | 51.25 | 93.45 | 52.4 | 37.41 |
+- **Repository:** https://github.com/huggingface/open-r1
+- **Training logs:** https://wandb.ai/huggingface/open-r1/runs/199cum6l
 
-In particular, on several categories of MT-Bench, Zephyr-7B-β has strong performance compared to larger open models like Llama2-Chat-70B:
+## Usage
 
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/6200d0a443eb0913fa2df7cc/raxvt5ma16d7T23my34WC.png)
+To chat with the model, first install 🤗 Transformers:
 
-However, on more complex tasks like coding and mathematics, Zephyr-7B-β lags behind proprietary models and more research is needed to close the gap.
-
-
-## Intended uses & limitations
+```shell
+pip install transformers>0.52
+```
 
-The model was initially fine-tuned on a filtered and preprocessed of the [`UltraChat`](https://huggingface.co/datasets/stingning/ultrachat) dataset, which contains a diverse range of synthetic dialogues generated by ChatGPT. 
-We then further aligned the model with [🤗 TRL's](https://github.com/huggingface/trl) `DPOTrainer` on the [openbmb/UltraFeedback](https://huggingface.co/datasets/openbmb/UltraFeedback) dataset, which contains 64k prompts and model completions that are ranked by GPT-4. As a result, the model can be used for chat and you can check out our [demo](https://huggingface.co/spaces/HuggingFaceH4/zephyr-chat) to test its capabilities. 
+Then run the chat CLI as follows:
 
-You can find the datasets used for training Zephyr-7B-β [here](https://huggingface.co/collections/HuggingFaceH4/zephyr-7b-6538c6d6d5ddd1cbb1744a66)
+```shell
+transformers chat open-r1/OpenR1-Distill-7B \
+    max_new_tokens=2048 \
+    do_sample=True \
+    temperature=0.6 \
+    top_p=0.95
+```
 
-Here's how you can run the model using the `pipeline()` function from 🤗 Transformers:
+Alternatively, run the model using the `pipeline()` function:
 
 ```python
-# Install transformers from source - only needed for versions <= v4.34
-# pip install git+https://github.com/huggingface/transformers.git
-# pip install accelerate
-
 import torch
 from transformers import pipeline
 
-pipe = pipeline("text-generation", model="HuggingFaceH4/zephyr-7b-beta", torch_dtype=torch.bfloat16, device_map="auto")
+pipe = pipeline("text-generation", model="open-r1/OpenR1-Distill-7B", torch_dtype=torch.bfloat16, device_map="auto")
 
-# We use the tokenizer's chat template to format each message - see https://huggingface.co/docs/transformers/main/en/chat_templating
 messages = [
-    {
-        "role": "system",
-        "content": "You are a friendly chatbot who always responds in the style of a pirate",
-    },
-    {"role": "user", "content": "How many helicopters can a human eat in one sitting?"},
+    {"role": "user", "content": "Which number is larger, 9.9 or 9.11?"},
 ]
-prompt = pipe.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
-outputs = pipe(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
+outputs = pipe(messages, max_new_tokens=2048, do_sample=True, temperature=0.6, top_p=0.95, return_full_text=False)
 print(outputs[0]["generated_text"])
-# <|system|>
-# You are a friendly chatbot who always responds in the style of a pirate.</s>
-# <|user|>
-# How many helicopters can a human eat in one sitting?</s>
-# <|assistant|>
-# Ah, me hearty matey! But yer question be a puzzler! A human cannot eat a helicopter in one sitting, as helicopters are not edible. They be made of metal, plastic, and other materials, not food!
 ```
 
-## Bias, Risks, and Limitations
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+## Performance
+
+We use [Lighteval](https://github.com/huggingface/lighteval) to evaluate models on the following benchmarks:
+
+| Model                       | AIME 2024 | MATH-500 | GPQA-D | LiveCodeBench |
+|-----------------------------|-----------|----------|--------|---------------|
+| OpenR1-Distill-7B           | 52.7      | 89.0     | 52.8   | 39.4          |
+| DeepSeek-R1-Distill-Qwen-7B | 51.3      | 93.5     | 52.4   | 37.4          |
+
+All scores denote pass@1 accuracy and use sampling with `temperature=0.6` and `top_p=0.95`. The DeepSeek-R1 tech report uses sampling with 4-64 responses per query to estimate pass@1, but does not specify the specific number of responses per benchmark. In the table above, we estimate pass@1 accuracy with the following number of responses per query:
+
+|   Benchmark   | Number of responses per query |
+|:-------------:|:-----------------------------:|
+|   AIME 2024   |              64               |
+|   MATH-500    |               4               |
+| GPQA Diamond  |               8               |
+| LiveCodeBench |              16               |
 
-Zephyr-7B-β has not been aligned to human preferences for safety within the RLHF phase or deployed with in-the-loop filtering of responses like ChatGPT, so the model can produce problematic outputs (especially when prompted to do so). 
-It is also unknown what the size and composition of the corpus was used to train the base model (`mistralai/Mistral-7B-v0.1`), however it is likely to have included a mix of Web data and technical sources like books and code. See the [Falcon 180B model card](https://huggingface.co/tiiuae/falcon-180B#training-data) for an example of this.
+Note that for benchmarks like AIME 2024, it is important to sample many responses as there are only 30 problems and this introduces high variance across repeated runs. The choice of how many responses to sample per prompt likely explains the small differences between our evaluation results and those reported by DeepSeek. Check out the [`open-r1` repo](https://github.com/huggingface/open-r1?tab=readme-ov-file#evaluating-models) for instructions on how to reproduce these results.
 
+## Training methodology
 
-## Training and evaluation data
+OpenR1-Distill-7B was trained using supervised fine-tuning (SFT) on the [Mixture-of-Thoughts](https://huggingface.co/datasets/open-r1/Mixture-of-Thoughts) dataset, which contains reasoning traces distilled from [DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1). To optimise the data mixture, we followed the same methodology described in the [Phi-4-reasoning tech report](https://huggingface.co/papers/2504.21318), namely that mixtures can be optimised independently per domain, and then combined into a single dataset. The figure below shows evolution of our experiments on the math and code domains:
 
+<img src="data_mixture.png" alt="Centered Image" style="display: block; margin: 0 auto;">
+
+The individual experiments correspond to the following:
+
+* exp1 - exp3: extending the model's base RoPE frequency from 10k to 100k, 200k, and 300k respectively.
+* exp4 - exp6: scaling the learning rate on the math and code mixtures from 1e-5 to 2e-5, and 4e-5 respectively.
+* exp7 - exp8: measuring the impact of sequence packing (exp7) versus no packing (exp8) on the math mixture.
+* exp9 - exp10: measuring the impact of training on all three mixtures (math, code, and science) versus training on math and code only.
 
 ### Training hyperparameters
 
 The following hyperparameters were used during training:
-- learning_rate: 5e-07
+
+- num_epochs: 5.0
+- learning_rate: 4.0e-05
+- num_devices: 8
 - train_batch_size: 2
-- eval_batch_size: 4
+- gradient_accumulation_steps: 8
+- total_train_batch_size: 2 * 8 * 8 = 128
 - seed: 42
-- distributed_type: multi-GPU
-- num_devices: 16
-- total_train_batch_size: 32
-- total_eval_batch_size: 64
+- distributed_type: DeepSpeed ZeRO-3
 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: linear
-- lr_scheduler_warmup_ratio: 0.1
-- num_epochs: 3.0
+- lr_scheduler_type: cosine_with_min_lr with min_lr_rate=0.1
+- lr_scheduler_warmup_ratio: 0.03
+- max_grad_norm: 0.2
 
 ### Training results
 
+During training, we monitor progress on AIME 2024, GPQA Diamond, and LiveCodeBench v4 every epoch. We use LiveCodeBench v4 to accelerate evaluation as it contains fewer problems than v5, yet is still representative of the full benchmark. The following plot shows the training results:
+
+<img src="train_results.png" alt="Centered Image" style="display: block; margin: 0 auto;">
 
 ### Framework versions
 
-- Transformers 4.35.0.dev0
-- Pytorch 2.0.1+cu118
-- Datasets 2.12.0
-- Tokenizers 0.14.0
+- Platform: Linux-5.15.0-1049-aws-x86_64-with-glibc2.31
+- Python version: 3.11.11
+- TRL version: 0.18.0.dev0
+- PyTorch version: 2.6.0
+- Transformers version: 4.52.0.dev0
+- Accelerate version: 1.4.0
+- Datasets version: 3.5.1
+- HF Hub version: 0.30.2
+- bitsandbytes version: 0.45.5
+- DeepSpeed version: 0.16.8
+- Liger-Kernel version: 0.5.9
+- OpenAI version: 1.76.2
+- vLLM version: 0.8.4
 
 ## Citation
+
+If you find this model is useful in your own work, please consider citing it as follows:
+
+```bibtex
+@misc{openr1,
+    title = {Open R1: A fully open reproduction of DeepSeek-R1},
+    url = {https://github.com/huggingface/open-r1},
+    author = {Hugging Face},
+    month = {January},
+    year = {2025}
+}
+```