Special Characters TrOCR
Is there a way to ignore the U+FFFD � replacement character when decoding generated ids?Screenshot 2023-05-13 at 2.15.511056×88 16.3 KB
2023-05-13T09:08:09Z
You can includebad_word_idsin thegeneratemethod to make sure a certain token is not generated:Utilities for Generation.
2023-05-13T12:40:42Z
Thank you for the reply. Seems it does not work for �
2023-05-14T05:59:15Z
Okay, I think i got the problem.� is shown when you form a character that does not actually exists.For example here i got  48820, 48820 (repeated two times instead that once) and that is whyi get the �.tensor([[    0,     0, 49173, 48247, 46311, 23133,  1437, 48827, 48820, 48820,23171, 49188, 27969, 47783,  5782, 48718, 48718, 27969, 27969, 48718,47780, 46311,  3602,  3602, 27819, 49045, 49045, 49045, 48897, 48897,9357,  9357,  9470,  9470,  9470,     2]], device=‘cuda:0’)Screenshot 2023-05-14 at 15.00.10848×58 4.13 KBIs there anything I can do to prevent token repetition? Or there is something I am doing wrong?