What is a "word"?
Trying to understand thechar_to_wordmethod ontransformers.BatchEncoding. The description in the docs is:Get the word in the original string corresponding to a character in the original string of a sequence of the batch.Is a “word” defined to be a collection of nonwhitespace characters mapped, by the tokenizer, to a single token?
2021-11-09T18:13:18Z
The notion of word depends on the tokenizer, and the text words are the result of the pre-tokenziation operation. Depending on the tokenizer, it can be split by whitespace, or by whitespace and punctuation, or other more advanced stuff
2021-11-09T18:55:27Z
Does each tokenizer overridechar_to_word? Is this even possible, sincechar_to_wordis defined onBatchEncoding? How do the subtleties of each tokenizer get transmitted tochar_to_word?
2021-11-09T19:39:49Z
TheBatchEncodinghas the correspondence from chars/tokens to words stored inside of it, which the fast tokenizer put there.
2021-11-09T19:52:24Z
Is this theoffset_mapping?
2021-11-09T19:58:45Z
No, it’s theoffset_mappingis the map token to span of characters. You need to combine it with theword_idsto get the map char to word.