File size: 1,251 Bytes
eb2d0f0 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
---
license: other
license_name: modified-mit
library_name: mlx
base_model: moonshotai/Kimi-K2-Instruct
pipeline_tag: text-generation
tags:
- mlx
---
<div>
<p style="margin-bottom: 0; margin-top: 0;">
<strong>See Kimi-K2 Dynamic MLX in action - <a href="https://youtu.be/-zfUvA2CDqE">https://youtu.be/-zfUvA2CDqE</a></strong>
</p>
<p style="margin-top: 0;margin-bottom: 0;">
<em>q3.95bit dynamic quant achieves 1.243 perplexity in our testing, slotting closer to q4 perplexity (1.168) than q3 perplexity (1.900).</em>
<div align="center">
| | |
|:---:|:---:|
| **q2** | perplexity: 41.293 |
| **q3** | perplexity: 1.900 |
| **q3.95** | perplexity: 1.243 |
| **q4** | perplexity: 1.168 |
| **q6** | perplexity: 1.128 |
| **q8** | perplexity: 1.128 |
</div>
</p>
</div>
<h1 style="margin-top: 0rem;">Kimi K2 Usage Notes</h1>
- Built with a modified version of MLX 0.26
- Runs on a single M3 Ultra 512GB RAM
- Requires expanding VRAM limit to at least ~500000 MB (I use 507000 for a larger context window)
<em>sudo sysctl iogpu.wired_limit_mb=507000</em>
- Expect ~20 tokens/s
- For more details see <a href="https://youtu.be/-zfUvA2CDqE">demonstration video</> or visit <a href="https://moonshotai.github.io/Kimi-K2/">Kimi K2</a>.
--- |