Update README.md
Browse files
README.md
CHANGED
@@ -14,7 +14,7 @@ tags:
|
|
14 |
|
15 |
日本語版は近日公開予定です(日本語を勉強中なので、間違いはご容赦ください!)
|
16 |
|
17 |
-
fio-base-japanese-v0.1 is a proof of concept, and the first release of the Fio family of Japanese embeddings. It is based on [cl-tohoku/bert-base-japanese-v3](https://huggingface.co/cl-tohoku/bert-base-japanese-v3) and trained on limited volumes of data on single GPU.
|
18 |
|
19 |
For more information, please refer to [my notes on Fio](https://ben.clavie.eu/fio).
|
20 |
|
@@ -50,7 +50,18 @@ Italic denotes best model for its size when a smaller model outperforms a bigger
|
|
50 |
| text-embedding-ada-002 | 0.790 | 0.789 | 0.7232 | 0.768 |
|
51 |
|
52 |
|
53 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
54 |
|
55 |
This model is best used through [sentence-transformers](https://www.SBERT.net). If you don't have it, it's easy to install:
|
56 |
|
@@ -70,10 +81,6 @@ print(embeddings)
|
|
70 |
```
|
71 |
|
72 |
|
73 |
-
## Usage
|
74 |
-
|
75 |
-
If using for a retrieval task, you must prefix your query with `"関連記事を取得するために使用できるこの文の表現を生成します: "`.
|
76 |
-
|
77 |
### Usage (HuggingFace Transformers)
|
78 |
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
|
79 |
|
|
|
14 |
|
15 |
日本語版は近日公開予定です(日本語を勉強中なので、間違いはご容赦ください!)
|
16 |
|
17 |
+
fio-base-japanese-v0.1 is a proof of concept, and the first release of the Fio family of Japanese embeddings. It is based on [cl-tohoku/bert-base-japanese-v3](https://huggingface.co/cl-tohoku/bert-base-japanese-v3) and trained on limited volumes of data on a single GPU.
|
18 |
|
19 |
For more information, please refer to [my notes on Fio](https://ben.clavie.eu/fio).
|
20 |
|
|
|
50 |
| text-embedding-ada-002 | 0.790 | 0.789 | 0.7232 | 0.768 |
|
51 |
|
52 |
|
53 |
+
|
54 |
+
## Usage
|
55 |
+
|
56 |
+
This model requires both `fugashi` and `unidic-lite`:
|
57 |
+
|
58 |
+
```
|
59 |
+
pip install -U fugashi unidic-lite
|
60 |
+
```
|
61 |
+
|
62 |
+
If using for a retrieval task, you must prefix your query with `"関連記事を取得するために使用できるこの文の表現を生成します: "`.
|
63 |
+
|
64 |
+
### Usage (Sentence-Transformers)
|
65 |
|
66 |
This model is best used through [sentence-transformers](https://www.SBERT.net). If you don't have it, it's easy to install:
|
67 |
|
|
|
81 |
```
|
82 |
|
83 |
|
|
|
|
|
|
|
|
|
84 |
### Usage (HuggingFace Transformers)
|
85 |
Without [sentence-transformers](https://www.SBERT.net), you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
|
86 |
|