Concatenate Sentances
Is there a helper class somewhere in one of the huggingface libraries which can assist in the concatenation of multiple short sentences into a single tensor - I want to tokenise each input example, shuffle and then combine until they’re ~512 tokens long. I thought that was part of the RoBERTa training process but I could be mistaken.