100% training WER on custom data
It is normal to get 100% WER for first few epochs. How many epochs did train? Also, there might be a mismatch between the datasets. For example, is the sampling rate in your data the same as it is in the LibriSpeech dataset? around 50 epoch, after 50 epochs its crashing
> It is normal to get 100% WER for first few epochs. How many epochs did train?

 > Also, there might be a mismatch between the datasets. For example, is the sampling rate in your data the same as it is in the LibriSpeech dataset?

sample rate is same but when I make a test run on lesser data i.e. around 200 hours at that time it was decreasing



![image](https://user-images.githubusercontent.com/46400775/115661535-6c700c80-a2f2-11eb-8a2d-bdfbb7b5b6b4.png)


Blue = ~200hours data
pink = ~400 hours data 60s is unnecessarily long. We typically use max 16.7s. Btw, by default, it may ignore everything longer than that.
Such long fragments doesn't help with training too much but will require large amount of GPU ram.

Also, what is your character set in your transcriptions? For QuartzNet, it is lowercase En letters and apostrophe ('), no punctuation or numbers.

Try lowering your learning rate and increasing batch size too. Several things - disable spec augment, reduce Weight Decay to 0 and see if the model can learn with no regularization first. 
The fact that 200 hours works and 400 doesn't means that your learning rate is too high - with 200 hours you have roughly half the number of steps per epoch so warmup and all other parameters should be adjusted. 
Hello,

I am finetuning the QuartznetBase-en5x5 model on my own dataset, but I am getting  trainingWER  100% and training loss is flatted after few steps although val_WER  and val_loss is decreasing gradually

**Dataset Details**
train Dataset consists of 169630 files totaling 410.18 hours
Val_ Dataset consists of   25036 files totaling 84.82 hours
Audio data is of length between( 5-60secs) and made sure that there are no samples without text output

I'm using the file [speech_to_text.py](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/speech_to_text.py) just made a little changes and added the following line before  _'trainer.fit'_


` asr_model.restore_from("QuartzNet15x5Base-En.nemo")`


**other details are:**
        trainer.max_epochs=200\
        model.train_ds.batch_size=16\
        model.validation_ds.batch_size=16 \
        model.optim.lr=0.00015 \
        +model.validation_ds.num_workers=2  \
        +model.train_ds.num_workers=10 \
        +trainer.precision=16 


**nemo version = 1.0.0rc1**

. Could you please troubleshoot where could the problem be?