getting out of memory error for finetuning
I am finetuning existing Citrinet model (stt_en_citrinet_1024). We have around 130h of transcribed T-mobile data.
The test and evaluation sets are fixed: test(3.53h) and evaluation(5.45h). I can't use all the train data because of out of memory error. This is the type of error that I'm getting: 
`RuntimeError: CUDA out of memory. Tried to allocate 434.00 MiB (GPU 0; 39.59 GiB total capacity; 37.22 GiB already allocated; 356.19 MiB free; 37.44 GiB reserved in total by PyTorch)`
The most amount of data that I could use was (90.29h) of data (max_duration = 65s).
I am using 4gpus (40g each). I reduced the batch_size to 16 and precision to 16.
I also, used the ctc-segmentation to split long audio files and the corresponding transcripts into shorter fragments. So, instead of 90h of data I can use 66h of data most with max_duration = 66. (not that much improvement on increasing the train size)
Any suggestion about how I can increase the train data? ( we are still not covering 1/3 of data)