Big dataset when being tokenized using map function gives type error as TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]
As stated above , I have a big dataset that is taking 2 days to tokenize . I am finetuning llama 3.1 and around 6 hrs in run into following errorTypeError                                 Traceback (most recent call last)in <cell line: 1>()----> 1 tokenized_datasets = ds.map(tokenize_function, batched=True)11 frames/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_fast.py in _batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, split_special_tokens)526             self._tokenizer.encode_special_tokens = split_special_tokens527→ 528         encodings = self._tokenizer.encode_batch(529             batch_text_or_text_pairs,530             add_special_tokens=add_special_tokens,TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]here is the code :from datasets import load_datasetds = load_dataset(“talkmap/telecom-conversation-corpus”)def tokenize_function(examples):return tokenizer(examples[“text”],padding=“max_length”,truncation=True,is_split_into_words=True)tokenized_datasets = ds.map(tokenize_function, batched=True)any help is appreciated