Does thoes lines in the equivalent manner in bert?
do those lines do or equivalent to each other in berttokenized= list(tokenizer.tokenize(sent))
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized)
tokens_tensor = torch.tensor([indexed_tokens])=input_ids = tokenizer.encode(sent)[None, :]