wordLevel tokenizing
hey guys,I want to tokenize a DNA seq. it’s like [[AAA GGG TTT],[AGT ATT CGC CCC AAA GTT],…]as you see it encompasses different sequences with different lengths, generally we have 64 unique codons (each of the elements in our list(for example AAA) is a codon)how can I tokenize in word level that separated only in spaces.also I  want to padding  and truncating  my tokenized list (with input_ids, mask columns that every tokenized file in hugging face pretrained library hassomehow this code works but it was not whta i wantfrom tokenizers import Tokenizerfrom tokenizers.models import WordLevelfrom tokenizers.trainers import WordLevelTrainerfrom tokenizers.pre_tokenizers import Whitespacefrom transformers import PreTrainedTokenizerFastCreate the tokenizertokenizer = Tokenizer(WordLevel(unk_token=“[UNK]”))tokenizer.pre_tokenizer = Whitespace()Define special tokensspecial_tokens_dict = {‘cls_token’: ‘[CLS]’, ‘sep_token’: ‘[SEP]’, ‘pad_token’: ‘[PAD]’, ‘mask_token’: ‘[MASK]’}tokenizer.add_special_tokens(special_tokens_dict)Define the trainer with special tokenstrainer = WordLevelTrainer(special_tokens=list(special_tokens_dict.values()))Example training datatraining_data = s.DNA_Codons.keys()Save the training data to a temporary filewith open(“training_data.txt”, “w”) as f:f.write(“\n”.join(training_data))Train the tokenizertokenizer.train([“training_data.txt”], trainer)Save the tokenizer to a filetokenizer.save(“C:\Users\farsh\Downloads\tokenizer_DNA.json”)