Update README.md

README.md

@@ -11,47 +11,11 @@ tags:
 - princeton-nlp/Llama-3-8B-ProLong-64k-Instruct
 ---
 
-# Llama-3-15B-Instruct-64k
+# Llama-3-15B-64k-Instruct
 
-Llama-3-15B-Instruct-64k is a merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
-* [princeton-nlp/Llama-3-8B-ProLong-64k-Instruct](https://huggingface.co/princeton-nlp/Llama-3-8B-ProLong-64k-Instruct)
-* [princeton-nlp/Llama-3-8B-ProLong-64k-Instruct](https://huggingface.co/princeton-nlp/Llama-3-8B-ProLong-64k-Instruct)
-* [princeton-nlp/Llama-3-8B-ProLong-64k-Instruct](https://huggingface.co/princeton-nlp/Llama-3-8B-ProLong-64k-Instruct)
-* [princeton-nlp/Llama-3-8B-ProLong-64k-Instruct](https://huggingface.co/princeton-nlp/Llama-3-8B-ProLong-64k-Instruct)
+I decided to repeat [this](https://huggingface.co/elinas/Llama-3-15B-Instruct-zeroed) merge, but using [64K version of Llama 3 8B](https://huggingface.co/princeton-nlp/Llama-3-8B-ProLong-64k-Instruct).
 
-## 🧩 Configuration
-
-```yaml
-dtype: bfloat16
-merge_method: passthrough
-slices:
-- sources:
-  - layer_range: [0, 24]
-    model: princeton-nlp/Llama-3-8B-ProLong-64k-Instruct
-- sources:
-  - layer_range: [8, 24]
-    model: princeton-nlp/Llama-3-8B-ProLong-64k-Instruct
-    parameters:
-      scale:
-      - filter: o_proj
-        value: 0.0
-      - filter: down_proj
-        value: 0.0
-      - value: 1.0
-- sources:
-  - layer_range: [8, 24]
-    model: princeton-nlp/Llama-3-8B-ProLong-64k-Instruct
-    parameters:
-      scale:
-      - filter: o_proj
-        value: 0.0
-      - filter: down_proj
-        value: 0.0
-      - value: 1.0
-- sources:
-  - layer_range: [24, 32]
-    model: princeton-nlp/Llama-3-8B-ProLong-64k-Instruct
-```
+This should work with a context up to 64k, but I strongly recommend making a finetune first.
 
 ## 💻 Usage
 
@@ -62,7 +26,7 @@ from transformers import AutoTokenizer
 import transformers
 import torch
 
-model = "Ttimofeyka/Llama-3-15B-Instruct-64k"
+model = "Ttimofeyka/Llama-3-15B-64k-Instruct"
 messages = [{"role": "user", "content": "What is a large language model?"}]
 
 tokenizer = AutoTokenizer.from_pretrained(model)