std::out_of_range error when training TransformerXL model
**Describe the bug**

When training an TransformerXL model by following the script transformer_lm.py, I get the following error:

![image](https://user-images.githubusercontent.com/10922796/174274040-4e21f930-425f-4ad5-95eb-4f92aa8b913a.png)

For this, I use the same train, eval and test json files as I use for the CTC conformer models which train fine. I use this in combination with a tokenizer trained with the following command:

`python /workspace/NeMo/scripts/process_asr_text_tokenizer.py --manifest=/workspace/NeMo/data/DataFrames/Baseline/Train/train.json --data_root=/workspace/Tokenizer_AllData_1024 --vocab_size=1024 --tokenizer="spe" --spe_type="unigram" --spe_character_coverage=1.0 --no_lower_case --log`

Again, the conformer CTC trained fine with this tokenizer.

Since this doesn't seem to be a python error, I can't figure out where it goes wrong.

Cant these tokenizers be used in combination with transformerXL?

Tokenizer config:

![image](https://user-images.githubusercontent.com/10922796/174276395-d557f3f9-246b-4880-a63f-0dc13aaa95ca.png)


The goal is to do ASR -> Beam search N-Gram -> TransformerXL

**Steps/Code to reproduce bug**

Build tokenizer -> change paths in conf/transformer_lm_config.yaml -> python transformer_lm.py

**Expected behavior**

TransformerXL model trains without problems

**Environment overview (please complete the following information)**
```

docker run --gpus all -it --shm-size=8g -p 8888:8888 -p 0.0.0.0:6006:6006 --name TransformerXL --ulimit memlock=-1 --ulimit stack=67108864 nvcr.io/nvidia/pytorch:22.05-py3

Installed NeMo with git clone and ./reinstall.sh
```

Python 3.8.13
Torch: 1.12.0a0+8a1a93a

