Is_split_into_words in encode_plus
See code,transformers/tokenization_utils.py at ae54e3c3b18bac0832ad62ea9b896dfd52a09850 · huggingface/transformers · GitHubIn _encode_plus function, it invokes self.tokenize(t, is_split_into_words=True, **kwargs), but tokenize() doesn’t use is_split_into_words.See code,github.comhuggingface/transformers/blob/ae54e3c3b18bac0832ad62ea9b896dfd52a09850/src/transformers/tokenization_utils.py#L481Whether the number of added tokens should be computed in the case of a sequence pair or a singlesequence.Returns:`int`: Number of special tokens added to sequences."""token_ids_0 = []token_ids_1 = []return len(self.build_inputs_with_special_tokens(token_ids_0, token_ids_1 if pair else None))def tokenize(self, text: TextInput, **kwargs) -> List[str]:"""Converts a string in a sequence of tokens, using the tokenizer.Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies(BPE/SentencePieces/WordPieces). Takes care of added tokens.Args:text (`str`):The sequence to be encoded.**kwargs (additional keyword arguments):It’s very weird.