 Compute SE in float32 for stability in AMP training.
~~Can you open a new PR?~~ nvm. I'm looking forward for this!  I'm hoping to train a citrinet model from scratch soon on A100 GPUs which could leverage really well mix-precision training.  Is there an ETA for this patch to land in the main trunk? As discussed with Jason and Som, we're leaving the workaround for the SE with AMP right now. Once I get to repro the exact issue, we can follow up with PyTorch team.

@jprobichaud, I am a bit absorbed by watching GTC this week :) 
Hopefully, the patch will land by the beginning of the next week.
Squeeze-and-Excitation block introduced instabilities to Citrinet when training with mixed precision.
This change avoids computing SE in float16 all together.

As SE block does not take much of computational time, the time and memory benefits of other computations done in float16 can allow to train model faster.  

The runs on NGC and multinode cluster confirm that there is no performance degradation or instabilities when using fp16 with current patch. The speedup is 15% and memory savings are around 3Gb for 21M parameter model.