Conformer : Training ASR for multiple language 
**What i did**

I took a multilingual SSL pretrained model and wanted to fine-tune it for ASR task on english and hindi together at once. I used a tokenizer that contains both English and Hindi manifest. 

**ERROR** 
After few steps the loss goes nan , detected inf or nan values in gradients! Setting gradients to zero.

**What i tried**
All of the hyper-parameters are kept as mentioned in GitHub repo.
i tried different combination of batch size and gradient accumulation. i tried lowering the learning rate.
____________________________________________________________
**AIM of my experiment**
let say the audio clip says : Aj mene apple khaya
want the transcription to be : आज मेने apple खाया
____________________________________________________________
how can this be achieved. 
Do i need to use the other method of tokenizing , where one make separate tokenizer for languages, and  use "agg" aggregate.
Can you suggest any method to achieve the above problem

An early response is always appreciated.
@titu1994 @meghmak13