What is a Text Pair in a tokenizer?
Hey all,I am relatively new to HuggingFace and deep NLP in general. I have noticed in the documentation and in some example notebooks I have seen that tokenizers are used as follows:tokenizer = SomeTokenizerClass()
encoding = tokenizer(text_to_tokenize, context_of_text)Wheretext_to_tokenizeandcontext_of_textare bothstrobjects. In the documentation, this type of call is shownhereWhat does this type of call to a tokenizer do and why would it be different thanencoding = tokenizer(text_to_tokenize + ' ' + context_of_text)Thank you so much for your help!
2022-09-03T15:58:33Z
Have you figured this out? I am also wondering the same. It seems there isn’t enough explanation on the Internet.
2023-07-18T23:11:36Z
tl;dr It adds a [SEP] (idx 102) between the two sentences.Here is your answer (see the long answer)BERT for multiple sentences - nlp - PyTorch Forums.