File size: 5,247 Bytes
d1611e6 2227833 d1611e6 2227833 d1611e6 2227833 9d3e41a 801a522 9d3e41a 801a522 9d3e41a 801a522 9d3e41a 801a522 9d3e41a 801a522 9d3e41a 801a522 9d3e41a 801a522 9d3e41a 801a522 9d3e41a 801a522 9d3e41a 801a522 9d3e41a 801a522 9d3e41a 801a522 9d3e41a 801a522 9d3e41a 801a522 9d3e41a 801a522 9d3e41a d1611e6 9d3e41a 2227833 9d3e41a 801a522 9d3e41a 801a522 9d3e41a 2227833 9d3e41a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 |
---
base_model:
- Qwen/Qwen2.5-Math-7B
library_name: transformers
license: mit
pipeline_tag: text-generation
tags:
- reasoning
- Zero-RL
---
# 📖Introduction
LUFFY is a reinforcement learning framework that bridges the gap between zero-RL and imitation learning by incorporating off-policy reasoning traces into the training process. Built upon GRPO, LUFFY combines on-policy rollouts with off-policy demonstrations during advantage estimation and introduces **policy shaping** via regularized importance sampling to emphasize low-probability yet crucial actions.
### Key Highlights:
- **Off-Policy Guidance:** Seamlessly integrates external reasoning traces to bootstrap learning from stronger models.
- **Dynamic Balance:** Learns when to imitate and when to explore, adapting over the course of training.
- **Policy Shaping:** Emphasizes important actions often ignored in standard policy gradients, enabling better generalization.
---
## Inference
Here’s an example of using LUFFY for inference:
```python
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
model_path="Elliott/LUFFY-Qwen-Math-7B-Zero"
question = "which number is larger? 9.11 or 9.9?"
tokenizer = AutoTokenizer.from_pretrained(model_path)
messages = [{"role": "user", "content": question}]
chat = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
llm = LLM(model=model_path)
params = SamplingParams(temperature=0.6, max_tokens=8192)
outputs = llm.generate([chat], params)
print(outputs[0].outputs[0].text)
```
---
# 📃Evaluation
LUFFY is evaluated on six competition-level benchmarks, achieving state-of-the-art results among all zero-RL methods. It surpasses both on-policy RL and imitation learning (SFT), especially in generalization:
| **Model** | **AIME 2024** | **AIME 2025** | **AMC** | **MATH-500** | **Minerva** | **Olympiad** | **Avg.** |
|-----------------------------------|-------------|-------------|---------|---------------|-------------|---------------|----------|
| Qwen2.5-Math | 12.9 | 4.2 | 32.6 | 48.8 | 10.7 | 14.8 | 20.7 |
| Qwen2.5-Math-Instruct | 11.4 | 8.8 | 48.3 | 81.2 | 33.1 | 38.8 | 36.9 |
| SimpleRL-Zero | 26.3 | 6.7 | 55.4 | 74.4 | 25.7 | 35.4 | 37.3 |
| OpenReasoner-Zero | 17.2 | 15.0 | 52.3 | 84.6 | 33.8 | 47.1 | 41.7 |
| PRIME-Zero | 17.9 | 14.7 | 55.2 | 79.4 | **38.2** | 42.2 | 41.3 |
| Oat-Zero | **31.7** | 11.0 | 61.6 | 79.2 | 29.8 | 42.5 | 42.6 |
| **LUFFY** | 29.5 | 23.2 | **66.1**| **88.4** | 33.8 | **56.4** | **49.6** |
---
LUFFY also generalizes well to out-of-distribution tasks, with over +6.2 average gain on ARC-C, GPQA, and MMLU-Pro.
| **Model** | **ARC-c** | **GPQA-diamond** | **MMLU-Pro** | **Avg.** |
|----------------------------------|-----------|------------------|--------------|----------|
| Qwen2.5-Math-7B-Base | 18.2 | 11.1 | 16.9 | 15.4 |
| Qwen2.5-Math-7B-Instruct | 70.3 | 24.7 | 34.1 | 43.0 |
| SimpleRL-Zero | 30.2 | 23.2 | 34.5 | 29.3 |
| PRIME-Zero | 73.3 | 18.2 | 32.7 | 41.4 |
| Oat-Zero | 70.1 | 23.7 | 41.7 | 45.2 |
| OpenReasoner-Zero | 66.2 | 29.8 | 58.7 | 51.6 |
| **LUFFY** | _80.5_ | _39.9_ | **53.0** | **57.8** |
---
# 🌻Acknowledgement
LUFFY builds upon [veRL](https://github.com/volcengine/verl) and [deepscaler](https://github.com/agentica-project/rllm), and utilizes [vLLM](https://github.com/vllm-project/vllm) for inference. We utilize [Math-Verify](https://github.com/huggingface/Math-Verify) for math reasoning evaluation. We thank the open-source community for datasets and backbones, including [NuminaMath](https://huggingface.co/datasets/AI-MO/NuminaMath-CoT), [OpenR1-Math-220k](https://huggingface.co/datasets/open-r1/OpenR1-Math-220k), [Qwen2.5-Math](https://github.com/QwenLM/Qwen2.5-Math), and [DeepSeek-R1](https://github.com/deepseek-ai/deepseek-r1) model.
Code: https://github.com/ElliottYan/LUFFY
# Citation
If you find our model, data, or evaluation code useful, please kindly cite our paper:
```bib
@misc{luffy,
title={Learning to Reason under Off-Policy Guidance},
author={Jianhao Yan and Yafu Li and Zican Hu and Zhi Wang and Ganqu Cui and Xiaoye Qu and Yu Cheng and Yue Zhang},
year={2025},
eprint={2504.14945},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2504.14945},
}
``` |