What does the parameter 'clean_up_tokenization_spaces' do in the tokenizer.decode function?
As titled. Thedocumentationis a bit vague.Whether or not to clean up the tokenization spaces.
2022-12-05T18:03:34Z
It should remove space artifacts inserted while encoding the sequence. E.g., if you havestate-of-the-artit will be encoded asstate - of - the - art. The cleanup should remove those spaces between-. Hope it helps!