Commit
·
2e769c2
1
Parent(s):
3ae2899
Update README.md
Browse files
README.md
CHANGED
|
@@ -15,6 +15,8 @@ widget:
|
|
| 15 |
- text: <|prompter|>Write a story about future of AI development</s><|assistant|>
|
| 16 |
---
|
| 17 |
|
|
|
|
|
|
|
| 18 |
This repo contains a low-rank adapter for **LLaMA-7b** fit on
|
| 19 |
- `Nebulous/gpt4all_pruned`
|
| 20 |
- `sahil2801/CodeAlpaca-20k`
|
|
@@ -35,20 +37,6 @@ This version of the weights was trained with the following hyperparameters:
|
|
| 35 |
The model was trained with flash attention and gradient checkpointing.
|
| 36 |
|
| 37 |
|
| 38 |
-
---
|
| 39 |
-
license: apache-2.0
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
# Open-Assistant SFT-1 12B Model
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
This is the first iteration English supervised-fine-tuning (SFT) model of
|
| 46 |
-
the [Open-Assistant](https://github.com/LAION-AI/Open-Assistant) project.
|
| 47 |
-
It is based on a Pythia 12B that was fine-tuned on ~22k human demonstrations
|
| 48 |
-
of assistant conversations collected through the
|
| 49 |
-
[https://open-assistant.io/](https://open-assistant.io/) human feedback web
|
| 50 |
-
app before March 7, 2023.
|
| 51 |
-
|
| 52 |
## Model Details
|
| 53 |
|
| 54 |
- **Developed** as part of the OpenAssistant Project
|
|
@@ -68,7 +56,7 @@ The input ends with the `<|assistant|>` token to signal that the model should
|
|
| 68 |
start generating the assistant reply.
|
| 69 |
|
| 70 |
|
| 71 |
-
|
| 72 |
|
| 73 |
```
|
| 74 |
from typing import List, NamedTuple
|
|
|
|
| 15 |
- text: <|prompter|>Write a story about future of AI development</s><|assistant|>
|
| 16 |
---
|
| 17 |
|
| 18 |
+
# LoRA Adapter for LLaMA 7B trained on more datasets than tloen/alpaca-lora-7b
|
| 19 |
+
|
| 20 |
This repo contains a low-rank adapter for **LLaMA-7b** fit on
|
| 21 |
- `Nebulous/gpt4all_pruned`
|
| 22 |
- `sahil2801/CodeAlpaca-20k`
|
|
|
|
| 37 |
The model was trained with flash attention and gradient checkpointing.
|
| 38 |
|
| 39 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
## Model Details
|
| 41 |
|
| 42 |
- **Developed** as part of the OpenAssistant Project
|
|
|
|
| 56 |
start generating the assistant reply.
|
| 57 |
|
| 58 |
|
| 59 |
+
##**Example Code** (Note several embeddings need to be loaded along with the LoRA weights):
|
| 60 |
|
| 61 |
```
|
| 62 |
from typing import List, NamedTuple
|