Add new SentenceTransformer model

README.md

@@ -1,127 +1,143 @@
 ---
 tags:
-- transformers
 - sentence-transformers
-language:
-- en
-license: apache-2.0
-library_name: transformers
-base_model:
-- RetroMAE
-model_index:
-- name: kpr-retromae
-results:
+- sentence-similarity
+- feature-extraction
+- dense
+pipeline_tag: sentence-similarity
+library_name: sentence-transformers
 ---
 
-# Knowledgeable Embedding: kpr-retromae
+# SentenceTransformer
 
-## Introduction
+This is a [sentence-transformers](https://www.SBERT.net) model trained. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
 
-**Injecting dynamically updatable entity knowledge into embeddings to enhance RAG**
+## Model Details
 
-A key limitation of large language models (LLMs) is their inability to capture less-frequent or up-to-date entity knowledge, often leading to factual inaccuracies and hallucinations. Retrieval-augmented generation (RAG), which incorporates external knowledge through retrieval, is a common approach to mitigate this issue.
+### Model Description
+- **Model Type:** Sentence Transformer
+<!-- - **Base model:** [Unknown](https://huggingface.co/unknown) -->
+- **Maximum Sequence Length:** 512 tokens
+- **Output Dimensionality:** 768 dimensions
+- **Similarity Function:** Dot Product
+<!-- - **Training Dataset:** Unknown -->
+<!-- - **Language:** Unknown -->
+<!-- - **License:** Unknown -->
 
-Although RAG typically relies on embedding-based retrieval, the embedding models themselves are also based on language models and therefore struggle with queries involving less-frequent entities, often failing to retrieve the crucial knowledge needed to overcome this limitation.
+### Model Sources
 
-**Knowledgeable Embedding** addresses this challenge by injecting real-world entity knowledge into embeddings, making them more *knowledgeable*.
+- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
+- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
+- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)
 
-**The entity knowledge is pluggable and can be dynamically updated.**
+### Full Model Architecture
 
-For further details, refer to [our paper](https://arxiv.org/abs/2507.03922) or [GitHub repository](https://github.com/knowledgeable-embedding/knowledgeable-embedding).
+```
+SentenceTransformer(
+  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'KPRModelForBert'})
+  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
+)
+```
 
-## Model List
+## Usage
 
-| Model | Model Size | Base Model |
-| --- | --- | --- |
-| [knowledgeable-ai/kpr-bert-base-uncased](https://huggingface.co/knowledgeable-ai/kpr-bert-base-uncased) | 112M | [bert-base-uncased](https://huggingface.co/google-bert/bert-base-uncased) |
-| [knowledgeable-ai/kpr-retromae](https://huggingface.co/knowledgeable-ai/kpr-retromae) | 112M | [RetroMAE](https://huggingface.co/Shitao/RetroMAE) |
-| [knowledgeable-ai/kpr-bge-base-en](https://huggingface.co/knowledgeable-ai/kpr-bge-base-en) | 112M | [bge-base-en](https://huggingface.co/BAAI/bge-base-en) |
-| [knowledgeable-ai/kpr-bge-base-en-v1.5](https://huggingface.co/knowledgeable-ai/kpr-bge-base-en-v1.5) | 112M | [bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) |
-| [knowledgeable-ai/kpr-bge-large-en-v1.5](https://huggingface.co/knowledgeable-ai/kpr-bge-large-en-v1.5) | 340M | [bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5) |
+### Direct Usage (Sentence Transformers)
 
-For practical use, we recommend `knowledgeable-ai/kpr-bge-*`, which significantly outperforms state-of-the-art models on queries involving less-frequent entities while performing comparably on other queries, as reported in [our paper](https://arxiv.org/abs/2507.03922).
+First install the Sentence Transformers library:
 
-Regarding the model size, we do not count the entity embeddings since they are stored in CPU memory and have a negligible impact on runtime performance. See [this page](https://github.com/knowledgeable-embedding/knowledgeable-embedding/wiki/Internals-of-Knowledgeable-Embedding) for details.
+```bash
+pip install -U sentence-transformers
+```
 
-## Model Details
+Then you can load this model and run inference.
+```python
+from sentence_transformers import SentenceTransformer
 
-- Model Name: kpr-retromae
-- Base Model: [RetroMAE](https://huggingface.co/Shitao/RetroMAE)
-- Maximum Sequence Length: 512
-- Embedding Dimension: 768
+# Download from the 🤗 Hub
+model = SentenceTransformer("knowledgeable-ai/kpr-retromae")
+# Run inference
+sentences = [
+    'The weather is lovely today.',
+    "It's so sunny outside!",
+    'He drove to the stadium.',
+]
+embeddings = model.encode(sentences)
+print(embeddings.shape)
+# [3, 768]
+
+# Get the similarity scores for the embeddings
+similarities = model.similarity(embeddings, embeddings)
+print(similarities)
+# tensor([[746.8777, 706.3154, 683.6250],
+#         [706.3154, 747.0701, 683.0114],
+#         [683.6249, 683.0115, 746.7446]])
+```
 
-## Usage
+<!--
+### Direct Usage (Transformers)
 
-This model can be used via [Hugging Face Transformers](https://github.com/huggingface/transformers) or [Sentence Transformers](https://github.com/UKPLab/sentence-transformers):
+<details><summary>Click to see the direct usage in Transformers</summary>
 
-### Hugging Face Transformers
+</details>
+-->
 
-```python
-from transformers import AutoTokenizer, AutoModel
-import torch
+<!--
+### Downstream Usage (Sentence Transformers)
 
-MODEL_NAME_OR_PATH = "knowledgeable-ai/kpr-retromae"
+You can finetune this model on your own dataset.
 
-input_texts = [
-    "Who founded Dominican Liberation Party?",
-    "Who owns Mompesson House?"
-]
+<details><summary>Click to expand</summary>
 
-# Load model and tokenizer from the Hugging Face Hub
-tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME_OR_PATH, trust_remote_code=True)
-model = AutoModel.from_pretrained(MODEL_NAME_OR_PATH, trust_remote_code=True)
+</details>
+-->
 
-# Preprocess the text
-preprocessed_inputs = tokenizer(input_texts, return_tensors="pt", padding=True)
+<!--
+### Out-of-Scope Use
 
-# Compute embeddings
-with torch.no_grad():
-    embeddings = model.encode(**preprocessed_inputs)
+*List how the model may foreseeably be misused and address what users ought not to do with the model.*
+-->
 
-print("Embeddings:", embeddings)
-```
+<!--
+## Bias, Risks and Limitations
 
-### Sentence Transformers
+*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
+-->
 
-```python
-from sentence_transformers import SentenceTransformer
+<!--
+### Recommendations
 
-MODEL_NAME_OR_PATH = "knowledgeable-ai/kpr-retromae"
+*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
+-->
 
-input_texts = [
-    "Who founded Dominican Liberation Party?",
-    "Who owns Mompesson House?"
-]
+## Training Details
 
-# Load model from the Hugging Face Hub
-model = SentenceTransformer(MODEL_NAME_OR_PATH, trust_remote_code=True)
+### Framework Versions
+- Python: 3.10.14
+- Sentence Transformers: 5.2.0.dev0
+- Transformers: 4.55.4
+- PyTorch: 2.4.0+cu121
+- Accelerate: 0.34.2
+- Datasets: 2.16.1
+- Tokenizers: 0.21.4
 
-# Compute embeddings
-embeddings = model.encode(input_texts)
+## Citation
 
-print("Embeddings:", embeddings)
-```
+### BibTeX
 
-**IMPORTANT:** This code will be supported in versions of Sentence Transformers later than v5.1.0, which have not yet been released at the time of writing. Until then, please install the library directly from GitHub:
+<!--
+## Glossary
 
-```bash
-pip install git+https://github.com/UKPLab/sentence-transformers.git
-```
+*Clearly define terms in order to be accessible across audiences.*
+-->
 
-## License
+<!--
+## Model Card Authors
 
-This model is licensed under the Apache License, Version 2.0.
+*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
+-->
 
-## Citation
+<!--
+## Model Card Contact
 
-If you use this model in your research, please cite the following paper:
-[Dynamic Injection of Entity Knowledge into Dense Retrievers](https://arxiv.org/abs/2507.03922)
-
-```bibtex
-@article{yamada2025kpr,
-title={Dynamic Injection of Entity Knowledge into Dense Retrievers},
-author={Ikuya Yamada and Ryokan Ri and Takeshi Kojima and Yusuke Iwasawa and Yutaka Matsuo},
-journal={arXiv preprint arXiv:2507.03922},
-year={2025}
-}
-```
+*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
+-->

entity_embeddings.npy

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bd83489d63bb45008620d90ba274331981546081491cfdd94be5afea9cb1cfea
+oid sha256:42f9795063bafacae304a2f79b362e44b4b9d5b4dd93b166528cee5129be8f63
 size 11126965376