Fix enja decoding
@mchrzanowski can you please send these changes to ``batch_translate`` branch? Will do at some point soon. don’t let me block you for now and merge away;
I’ll fix things in a subsequent pr later

On Mon, Mar 1, 2021 at 16:53 Oleksii Kuchaiev <notifications@github.com>
wrote:

> @mchrzanowski <https://github.com/mchrzanowski> can you please send these
> changes to batch_translate branch?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/NVIDIA/NeMo/pull/1820#issuecomment-788461636>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKER5ZWEGVZUMZOCBII5W3TBQZJHANCNFSM4YM72J6A>
> .
>
 i think this PR hopefully wont have to be merged. i think the path of least resistance for en<>ja will be to have the preprocessing pipeline produce integer ids as opposed to tokens. that way, there's no ambiguity between sequences of chars or tokens. detokenization will i think at that point just work with the default pathway and we wont have to special case en<>ja like this draft does.
the current en<>ja preprocessing pipeline is being shifted to using integers for intermediate steps. this will avoid tokenization issues between yttm, sentencepiece, and sacremoses due to parsing incomptabilities. this PR changes the detokenization to deal with integer inputs and output tokens only at the end.