[Question] Why using Learning Rate = 0.001 instead of 0.01 during fine-tuning ASR NeMo on non-English?
When we train base model (e.g. from which you fine-tune), we typically decay learning rate using cosine policy. The full lr plot thus looks something like this:
![image](https://user-images.githubusercontent.com/22035961/95894741-5e877080-0d3e-11eb-8c41-1304bb84a3a1.png)
Notice that at the end of the training it is 0 (or very near to zero).

When fine-tuning, you typically re-use the encoder weights (which is most of the network's weights) and therefore significantly increasing learning rate can cause divergence. The decoder, is randomly initialized for fine-tuning and would tolerate a bigger learning rate, but you have to balance that with encoder's needs - hence smaller lr is needed.

Using 10x smaller rate is the advice we found to work well empirically for several transfer learning applications. Please see this paper for experiments: https://arxiv.org/abs/2005.04290 
 @okuchaiev what is the default min-lr for the cosine scheduling? > @okuchaiev what is the default min-lr for the cosine scheduling?

By default it decays to 0 at the end of last epoch. You can change that using ``sched`` configuration section Thanks you @okuchaiev . You were right.
I changed the Learning Rate from 0.01 to 0.001 while using transfer learning and now WER had been decreased about 30%.
 
**Question**
According to #1107  which @okuchaiev suggest "**Try 0.001 (or even smaller). your learning rate is certainly too high. **" 
I want to know the reason why we should do that? 
I would appreciate if anyone can help me with that.

**Environment overview:**
-Environment location: Google Colab
-Method of NeMo install: !python -m pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[all]

**Environment details**
-OS version : Ubuntu 18.04
-PyTorch version : 1.6.0+cu101
-Python version : Python 3.6.9
-GPU model: Tesla T4