Update README.md
Browse files
README.md
CHANGED
|
@@ -50,7 +50,31 @@ For training data, we generate long contexts by augmenting [SlimPajama](https://
|
|
| 50 |
| GPU Type | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S |
|
| 51 |
| Minutes to Train (Wall)| 202 | 555 | 61 | 87 |
|
| 52 |
|
| 53 |
-
**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
- [GGUF](https://huggingface.co/crusoeai/Llama-3-8B-Instruct-1048k-GGUF)
|
| 55 |
- [MLX-4bit](https://huggingface.co/mlx-community/Llama-3-8B-Instruct-1048k-4bit)
|
| 56 |
|
|
|
|
| 50 |
| GPU Type | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S |
|
| 51 |
| Minutes to Train (Wall)| 202 | 555 | 61 | 87 |
|
| 52 |
|
| 53 |
+
**Evaluation:**
|
| 54 |
+
|
| 55 |
+

|
| 56 |
+
|
| 57 |
+
```
|
| 58 |
+
EVAL_MAX_CONTEXT_LENGTH=1040200
|
| 59 |
+
EVAL_MIN_CONTEXT_LENGTH=100
|
| 60 |
+
EVAL_CONTEXT_INTERVAL=86675
|
| 61 |
+
EVAL_DEPTH_INTERVAL=0.2
|
| 62 |
+
EVAL_RND_NUMBER_DIGITS=8
|
| 63 |
+
|
| 64 |
+
HAYSTACK1:
|
| 65 |
+
EVAL_GENERATOR_TOKENS=25
|
| 66 |
+
|
| 67 |
+
HAYSTACK2:
|
| 68 |
+
EVAL_CONTEXT_INTERVAL=173350
|
| 69 |
+
EVAL_GENERATOR_TOKENS=150000
|
| 70 |
+
|
| 71 |
+
HAYSTACK3:
|
| 72 |
+
EVAL_GENERATOR_TOKENS=925000
|
| 73 |
+
```
|
| 74 |
+
|
| 75 |
+
All boxes not pictured for Haystack 1 and 3 are 100% accurate. Haystacks 1,2 and 3 are further detailed in this [blog post](https://gradient.ai/blog/the-haystack-matters-for-niah-evals).
|
| 76 |
+
|
| 77 |
+
**Quants:**
|
| 78 |
- [GGUF](https://huggingface.co/crusoeai/Llama-3-8B-Instruct-1048k-GGUF)
|
| 79 |
- [MLX-4bit](https://huggingface.co/mlx-community/Llama-3-8B-Instruct-1048k-4bit)
|
| 80 |
|