--- language: - en license: apache-2.0 tags: - biencoder - sentence-transformers - text-classification - sentence-pair-classification - semantic-similarity - semantic-search - retrieval - reranking - generated_from_trainer - dataset_size:1056095 - loss:CoSENTLoss base_model: Alibaba-NLP/gte-modernbert-base widget: - source_sentence: In 2015 Adolf Hitler appeared in the kickstarter short movie `` Kung Fury `` as Taccone ( A.K.A . sentences: - In 2015 , Adolf Hitler appeared in the Kickstarter - short film `` Kung Fury `` as Taccone ( A.K.A . - In 1795 , the only white residents were Dr. John Laidley and two brothers with the surname Ainslie . - The 125th University Match was played in March 2014 at the Rye Golf Club , Oxford , East Sussex won the game 8.5 - 6.5 . - source_sentence: From 1973 to 1974 , Aubrey toured with the Cambridge Theatre Company as Diggory in `` She Stoops to Conquer `` and again as Aguecheek . sentences: - Oxide can be reduced to metallic samarium at higher temperatures by heating with a reducing agent such as hydrogen or carbon monoxide . - From 1973 to 1974 Aguecheek toured with the Cambridge Theatre Company as Diggory in `` You Stoops to Conquer `` and again as Aubrey . - The medals were presented by Barry Maister , IOC member , New Zealand and Sarah Webb Gosling , Vice President of World Sailing . - source_sentence: There is no official wall on the border , although there are sections of fence near populated areas and continuous border crossings . sentences: - The 2014 -- 15 Boston Bruins season was the 91st season for the National Hockey League franchise that was established on November 1 , 1924 . - He was trained by the Inghams and owned by John Hawkes . - There is no continuous wall on the border , although there are fence sections near populated areas and official border crossings . - source_sentence: Capital . `` The French established similar hill stations in Indochina , such as Dalat built in 1921 . sentences: - Lubuk China is a small town in Alor Gajah District , Melaka , Malaysia . It is situated near the border with Negeri Sembilan . - The French established similar hill stations in Indochina , such as Dalat , built in 1921 . - John Potts ( or Pott ) was a doctor and colonial governor of Virginia in the Jamestown settlement at Virginia Colony in the early 17th century . - source_sentence: The band pursued `` signals `` in January 2012 in three weeks , and drums were recorded in a day and a half . sentences: - It was repaired at the beginning of the 20th century and is listed as closed in our records . - The band tracked `` Signals `` in three weeks in January 2012 . Drums were recorded in a day and a half . - Contributors include actor Anton LaVey , Satanist Christopher Lee , serial killer expert Clive Barker , author Karen Greenlee , and necrophile Robert Ressler . datasets: - aditeyabaral-redis/langcache-sentencepairs-v1 pipeline_tag: sentence-similarity library_name: sentence-transformers metrics: - cosine_accuracy - cosine_accuracy_threshold - cosine_f1 - cosine_f1_threshold - cosine_precision - cosine_recall - cosine_ap - cosine_mcc model-index: - name: Redis fine-tuned BiEncoder model for semantic caching on LangCache results: - task: type: binary-classification name: Binary Classification dataset: name: val type: val metrics: - type: cosine_accuracy value: 0.7629982153480072 name: Cosine Accuracy - type: cosine_accuracy_threshold value: 0.8640064001083374 name: Cosine Accuracy Threshold - type: cosine_f1 value: 0.6907391673746814 name: Cosine F1 - type: cosine_f1_threshold value: 0.8261547684669495 name: Cosine F1 Threshold - type: cosine_precision value: 0.6290946608202218 name: Cosine Precision - type: cosine_recall value: 0.7657770800627943 name: Cosine Recall - type: cosine_ap value: 0.7350929175906749 name: Cosine Ap - type: cosine_mcc value: 0.47714361581572273 name: Cosine Mcc - task: type: binary-classification name: Binary Classification dataset: name: test type: test metrics: - type: cosine_accuracy value: 0.7035036519888425 name: Cosine Accuracy - type: cosine_accuracy_threshold value: 0.8520700931549072 name: Cosine Accuracy Threshold - type: cosine_f1 value: 0.7118460123901542 name: Cosine F1 - type: cosine_f1_threshold value: 0.8109649419784546 name: Cosine F1 Threshold - type: cosine_precision value: 0.5979034259235814 name: Cosine Precision - type: cosine_recall value: 0.8794413407821229 name: Cosine Recall - type: cosine_ap value: 0.6473553527394227 name: Cosine Ap - type: cosine_mcc value: 0.4408784752892243 name: Cosine Mcc --- # Redis fine-tuned BiEncoder model for semantic caching on LangCache This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [Alibaba-NLP/gte-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-modernbert-base) on the [LangCache Sentence Pairs (all)](https://huggingface.co/datasets/aditeyabaral-redis/langcache-sentencepairs-v1) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for sentence pair similarity. ## Model Details ### Model Description - **Model Type:** Sentence Transformer - **Base model:** [Alibaba-NLP/gte-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-modernbert-base) - **Maximum Sequence Length:** 8192 tokens - **Output Dimensionality:** 768 dimensions - **Similarity Function:** Cosine Similarity - **Training Dataset:** - [LangCache Sentence Pairs (all)](https://huggingface.co/datasets/aditeyabaral-redis/langcache-sentencepairs-v1) - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) ### Full Model Architecture ``` SentenceTransformer( (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'}) (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) ) ``` ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import SentenceTransformer # Download from the 🤗 Hub model = SentenceTransformer("aditeyabaral-redis/langcache-embed-v3") # Run inference sentences = [ 'The band pursued `` signals `` in January 2012 in three weeks , and drums were recorded in a day and a half .', 'The band tracked `` Signals `` in three weeks in January 2012 . Drums were recorded in a day and a half .', 'Contributors include actor Anton LaVey , Satanist Christopher Lee , serial killer expert Clive Barker , author Karen Greenlee , and necrophile Robert Ressler .', ] embeddings = model.encode(sentences) print(embeddings.shape) # [3, 768] # Get the similarity scores for the embeddings similarities = model.similarity(embeddings, embeddings) print(similarities) # tensor([[1.0000, 0.9599, 0.4944], # [0.9599, 1.0000, 0.5097], # [0.4944, 0.5097, 1.0000]]) ``` ## Evaluation ### Metrics #### Binary Classification * Datasets: `val` and `test` * Evaluated with [BinaryClassificationEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.BinaryClassificationEvaluator) | Metric | val | test | |:--------------------------|:-----------|:-----------| | cosine_accuracy | 0.763 | 0.7035 | | cosine_accuracy_threshold | 0.864 | 0.8521 | | cosine_f1 | 0.6907 | 0.7118 | | cosine_f1_threshold | 0.8262 | 0.811 | | cosine_precision | 0.6291 | 0.5979 | | cosine_recall | 0.7658 | 0.8794 | | **cosine_ap** | **0.7351** | **0.6474** | | cosine_mcc | 0.4771 | 0.4409 | ## Training Details ### Training Dataset #### LangCache Sentence Pairs (all) * Dataset: [LangCache Sentence Pairs (all)](https://huggingface.co/datasets/aditeyabaral-redis/langcache-sentencepairs-v1) * Size: 62,021 training samples * Columns: sentence1, sentence2, and label * Approximate statistics based on the first 1000 samples: | | sentence1 | sentence2 | label | |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:------------------------------------------------| | type | string | string | int | | details | | | | * Samples: | sentence1 | sentence2 | label | |:--------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------|:---------------| | The newer Punts are still very much in existence today and race in the same fleets as the older boats . | The newer punts are still very much in existence today and run in the same fleets as the older boats . | 1 | | Turner Valley , was at the Turner Valley Bar N Ranch Airport , southwest of the Turner Valley Bar N Ranch , Alberta , Canada . | Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada . | 0 | | After losing his second election , he resigned as opposition leader and was replaced by Geoff Pearsall . | Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall . | 1 | * Loss: [CoSENTLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters: ```json { "scale": 20.0, "similarity_fct": "pairwise_cos_sim" } ``` ### Evaluation Dataset #### LangCache Sentence Pairs (all) * Dataset: [LangCache Sentence Pairs (all)](https://huggingface.co/datasets/aditeyabaral-redis/langcache-sentencepairs-v1) * Size: 62,021 evaluation samples * Columns: sentence1, sentence2, and label * Approximate statistics based on the first 1000 samples: | | sentence1 | sentence2 | label | |:--------|:----------------------------------------------------------------------------------|:----------------------------------------------------------------------------------|:------------------------------------------------| | type | string | string | int | | details | | | | * Samples: | sentence1 | sentence2 | label | |:--------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------|:---------------| | The newer Punts are still very much in existence today and race in the same fleets as the older boats . | The newer punts are still very much in existence today and run in the same fleets as the older boats . | 1 | | Turner Valley , was at the Turner Valley Bar N Ranch Airport , southwest of the Turner Valley Bar N Ranch , Alberta , Canada . | Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada . | 0 | | After losing his second election , he resigned as opposition leader and was replaced by Geoff Pearsall . | Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall . | 1 | * Loss: [CoSENTLoss](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cosentloss) with these parameters: ```json { "scale": 20.0, "similarity_fct": "pairwise_cos_sim" } ``` ### Training Logs | Epoch | Step | val_cosine_ap | test_cosine_ap | |:-----:|:----:|:-------------:|:--------------:| | -1 | -1 | 0.7351 | 0.6474 | ### Framework Versions - Python: 3.12.3 - Sentence Transformers: 5.1.0 - Transformers: 4.55.0 - PyTorch: 2.8.0+cu128 - Accelerate: 1.10.0 - Datasets: 4.0.0 - Tokenizers: 0.21.4 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ``` #### CoSENTLoss ```bibtex @online{kexuefm-8847, title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT}, author={Su Jianlin}, year={2022}, month={Jan}, url={https://kexue.fm/archives/8847}, } ```