This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

• Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (2025)
• Reinforcement Learning Enhanced LLMs: A Survey (2024)
• rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking (2025)
• OpenRFT: Adapting Reasoning Foundation Model for Domain-specific Tasks with Reinforcement Fine-Tuning (2024)
• Diving into Self-Evolving Training for Multimodal Reasoning (2024)
• o1-Coder: an o1 Replication for Coding (2024)
• Mars-PO: Multi-Agent Reasoning System Preference Optimization (2024)

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

hi

No open weights??
If exists please share link.

Missing Citations: Kimi K1.5 Training Loss Function Mirrors SPPO/GPO Loss

I’ve been reviewing the training loss function presented in the Kimi K1.5 paper and have serious concerns regarding its originality. It appears that the loss function is nearly identical in form to the one introduced in the SPPO/GPO work. Specifically, both formulations involve a squared error between a log-probability ratio and a scaled reward (or preference score) term, along with a normalization factor. Here’s a side-by-side comparison:

GPO/SPPO Loss

(Excerpt from GPO Paper (https://arxiv.org/abs/2410.02197) and SPPO Paper (https://arxiv.org/abs/2405.00675))

$${\theta }_{t+1}=\arg \underset{\theta }{\min }{\mathbb{E}}_{\mathbf{x}\sim \mathcal{X},\mathbf{y}\sim {\pi }_{{\theta }_t}(\cdot \mid \mathbf{x})}\left[{(\log \frac{{\pi }_{\theta }(\mathbf{y}\mid \mathbf{x})}{{\pi }_{{\theta }_t}(\mathbf{y}\mid \mathbf{x})}-\frac{1}{\beta }(\widehat{s}(\mathbf{y}\succ {\pi }_{{\theta }_t}\mid \mathbf{x})-\log Z_{{\pi }_{{\theta }_t}}(\mathbf{x})\left)\right)}^2\right]\backslash theta\_\{t+1\}\; =\; \backslash arg\; \backslash min\_\{\backslash theta\}\; \backslash mathbb\{E\}\_\{\backslash mathbf\{x\}\; \backslash sim\; \backslash mathcal\{X\},\; \backslash mathbf\{y\}\; \backslash sim\; \backslash pi\_\{\backslash theta\_t\}(\backslash cdot\; \backslash mid\; \backslash mathbf\{x\})\}\; \backslash left[\; \backslash left(\; \backslash log\; \backslash frac\{\backslash pi\_\{\backslash theta\}(\backslash mathbf\{y\}\; \backslash mid\; \backslash mathbf\{x\})\}\{\backslash pi\_\{\backslash theta\_t\}(\backslash mathbf\{y\}\; \backslash mid\; \backslash mathbf\{x\})\}\; -\backslash frac\{1\}\{\backslash beta\}\; \backslash Big(\; \backslash widehat\{s\}\backslash left(\backslash mathbf\{y\}\; \backslash succ\; \backslash pi\_\{\backslash theta\_t\}\; \backslash mid\; \backslash mathbf\{x\}\backslash right)\; -\; \backslash log\; Z\_\{\backslash pi\_\{\backslash theta\_t\}\}(\backslash mathbf\{x\})\; \backslash Big)\; \backslash right)^2\; \backslash right]$$

Kimi K1.5 Loss

(Excerpt from the Kimi K1.5 Paper)

$$L(\theta )={\mathbb{E}}_{(x,y^{\ast })\sim \mathcal{D}}\text{\hspace{0.17em}⁣}\left[{\mathbb{E}}_{(y,z)\sim {\pi }_{{\theta }_i}}\text{\hspace{0.17em}⁣}\left[{(r(x,y,y^{\ast })-\tau \log Z-\tau \log \frac{{\pi }_{\theta }(y,z\mid x)}{{\pi }_{{\theta }_i}(y,z\mid x)})}^2\right]\right]L(\backslash theta)=\backslash mathbb\{E\}\_\{(x,\; y^*)\; \backslash sim\; \backslash mathcal\{D\}\}\backslash !\backslash left[\backslash mathbb\{E\}\_\{(y,\; z)\; \backslash sim\; \backslash pi\_\{\backslash theta\_i\}\}\backslash !\backslash left[\backslash left(r\backslash left(x,\; y,\; y^*\backslash right)-\backslash tau\; \backslash log\; Z\; -\; \backslash tau\; \backslash log\; \backslash frac\{\backslash pi\_\backslash theta(y,\; z\; \backslash mid\; x)\}\{\backslash pi\_\{\backslash theta\_i\}(y,\; z\; \backslash mid\; x)\}\backslash right)^2\backslash right]\backslash right]$$

While the notations differ slightly (e.g., $r(x,y,y^*)$ vs. $\widehat{s}\left(\mathbf{y} \succ \pi_{\theta_t} \mid \mathbf{x}\right)$ and $\tau$ vs. $1/\beta$, the structural similarity is striking. Both losses adjust the policy by matching a log-probability ratio to a reward (or preference score) signal, with a normalization constant to stabilize training.

Questions/Concerns:

1. Justification for Differences:
   If there are intended differences (e.g., in reward computation or sampling strategy), could the authors clearly delineate these differences?

2. Proper Attribution and Citation:
   The training loss function in Kimi K1.5 appears to directly mirror that of the SPPO/GPO work. Could the authors update the manuscript to include explicit citations to the original SPPO/GPO papers? Proper attribution is crucial to maintain academic integrity and give due credit for prior work.

Request for Clarification:

This issue is raised to ensure academic integrity and clarity on the contributions of the Kimi K1.5 paper. A prompt explanation or pointer to supplementary material addressing these concerns, including a revision of citations, would be greatly appreciated.

Thank you for your attention to this matter.