What is best way to compute document similarity?
What is the best way to compute document similarity?I was thinking to use SentenceTransformers for measuring document similarity.https://www.sbert.net/Is this is the best way?Also is there a model to apply contrastive learning for document similarity learning?
2022-06-21T20:20:32Z
Yup, SentenceTransformers can definitely be used for measuring document similarity. Depending on the size of your documents, you might want tochoose a modelthat was tuned for dot-product similarity. (E.g. fromthe MSMARCO docs: “Models with normalized embeddings will prefer the retrieval of shorter passages, while models tuned fordot-productwill prefer the retrieval of longer passages.”) You might also have to split large passages into chunks, otherwise content gets truncated for the models.For contrastive learning, I think you could usesentence-transformers/all-MiniLM-L6-v2 · Hugging FacewithContrastiveLoss.We’re actually looking at ways to improve the user experience with SentenceTransformers + Hugging Face, so feel free to post here or message me directly if you have any questions or feedback