Update README.md
Browse files
README.md
CHANGED
|
@@ -1,3 +1,42 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: mit
|
| 3 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
---
|
| 4 |
+
|
| 5 |
+
|
| 6 |
+
Please refer to the [SepLLM paper - ICML 2025](https://arxiv.org/abs/2412.12094) and our [`GitHub repository`](https://github.com/HKUDS/SepLLM) for using this model.
|
| 7 |
+
|
| 8 |
+
To use the checkpoint of this model, you must install the `transformers-4.38.0.post1+sepllm-py3-none-any.whl` released from our [`GitHub repository`](https://github.com/HKUDS/SepLLM). Below are the reference script for testing and a sample of test results. We conducted testing using `lm_eval==0.4.0`.
|
| 9 |
+
|
| 10 |
+
```
|
| 11 |
+
CUDA_LAUNCH_BLOCKING=1
|
| 12 |
+
lm_eval --model hf \
|
| 13 |
+
--model_args pretrained=Gausson/pythia-160m-deduped-n128-SepLLM \
|
| 14 |
+
--tasks arc_challenge,arc_easy,lambada_openai,logiqa,piqa,sciq,winogrande,wsc,wikitext \
|
| 15 |
+
--num_fewshot 5 \
|
| 16 |
+
--device cuda:0\
|
| 17 |
+
--batch_size 32
|
| 18 |
+
```
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
```
|
| 22 |
+
hf (pretrained=Gausson/pythia-160m-deduped-n128-SepLLM), gen_kwargs: (), limit: None, num_fewshot: 5, batch_size: 32
|
| 23 |
+
| Tasks |Version|Filter|n-shot| Metric | Value | |Stderr|
|
| 24 |
+
|--------------|-------|------|-----:|---------------|------:|---|-----:|
|
| 25 |
+
|arc_challenge |Yaml |none | 5|acc | 0.2014|± |0.0117|
|
| 26 |
+
| | |none | 5|acc_norm | 0.2346|± |0.0124|
|
| 27 |
+
|arc_easy |Yaml |none | 5|acc | 0.4731|± |0.0102|
|
| 28 |
+
| | |none | 5|acc_norm | 0.4520|± |0.0102|
|
| 29 |
+
|lambada_openai|Yaml |none | 5|perplexity |30.1605|± |1.0128|
|
| 30 |
+
| | |none | 5|acc | 0.3315|± |0.0066|
|
| 31 |
+
|logiqa |Yaml |none | 5|acc | 0.2273|± |0.0164|
|
| 32 |
+
| | |none | 5|acc_norm | 0.2857|± |0.0177|
|
| 33 |
+
|piqa |Yaml |none | 5|acc | 0.6464|± |0.0112|
|
| 34 |
+
| | |none | 5|acc_norm | 0.6447|± |0.0112|
|
| 35 |
+
|sciq |Yaml |none | 5|acc | 0.8260|± |0.0120|
|
| 36 |
+
| | |none | 5|acc_norm | 0.8150|± |0.0123|
|
| 37 |
+
|wikitext |Yaml |none | 5|word_perplexity|30.3488| | |
|
| 38 |
+
| | |none | 5|byte_perplexity| 1.8931| | |
|
| 39 |
+
| | |none | 5|bits_per_byte | 0.9207| | |
|
| 40 |
+
|winogrande |Yaml |none | 5|acc | 0.5178|± |0.0140|
|
| 41 |
+
|wsc |Yaml |none | 5|acc | 0.3750|± |0.0477|
|
| 42 |
+
```
|