mtp
#4
by CHNtentes - opened
could you add mtp in future models?
+1 on this
or if someone makes a eagle3 draft model for it, would be great
At the moment, we don’t have the bandwidth to build this version. Contributions and proposals from the community are very welcome.
Didn't you guys already train MTP heads during RL for this model? The config even has left over references to it. Could the weights with the MTP heads be released?
Quote from section 3.3 of your Forge blog post:
"MTP-based Speculative Decoding: Instead of static draft models, we use Multi-Token Prediction (MTP) heads continuously fine-tuned via Top-K KL loss. This ensures alignment with the evolving RL policy, sustaining high acceptance rates and significant speedup by mitigating distribution shifts."