Split document into sentences for sentence embedding
Hi,Wondering if there is a Huggingface alternative to the Gensim split_sentences method to take a document and split into sentences ready for model.encode()?https://www.kite.com/python/docs/gensim.summarization.textcleaner.split_sentencesA first timer says many thanks
2021-02-09T12:10:36Z
So you want to split a text into sentences and then create a sentence embedding for each sentence? Just use a parser like stanza or spacy to tokenize/sentence segment your data. This is typically the first step in many NLP tasks.
2021-02-09T12:26:29Z
Indeed, just wondered if Huggingface had their own variant. Thanks