File size: 1,251 Bytes
eb2d0f0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
---
license: other
license_name: modified-mit
library_name: mlx
base_model: moonshotai/Kimi-K2-Instruct
pipeline_tag: text-generation
tags:
- mlx
---
<div>
  <p style="margin-bottom: 0; margin-top: 0;">
    <strong>See Kimi-K2 Dynamic MLX in action - <a href="https://youtu.be/-zfUvA2CDqE">https://youtu.be/-zfUvA2CDqE</a></strong>
  </p>
<p style="margin-top: 0;margin-bottom: 0;">
    <em>q3.95bit dynamic quant achieves 1.243 perplexity in our testing, slotting closer to q4 perplexity (1.168) than q3 perplexity (1.900).</em>
    <div align="center">
| | |
|:---:|:---:|
| **q2** | perplexity: 41.293 |
| **q3** | perplexity: 1.900 |
| **q3.95** | perplexity: 1.243 |
| **q4** | perplexity: 1.168 |
| **q6** | perplexity: 1.128 |
| **q8** | perplexity: 1.128 |
</div>
  </p>
</div>

<h1 style="margin-top: 0rem;">Kimi K2 Usage Notes</h1>
- Built with a modified version of MLX 0.26
- Runs on a single M3 Ultra 512GB RAM
- Requires expanding VRAM limit to at least ~500000 MB (I use 507000 for a larger context window)
<em>sudo sysctl iogpu.wired_limit_mb=507000</em>
- Expect ~20 tokens/s
- For more details see <a href="https://youtu.be/-zfUvA2CDqE">demonstration video</> or visit <a href="https://moonshotai.github.io/Kimi-K2/">Kimi K2</a>.
---