Rethinking the role of normalization and residual blocks for spiking
  neural networks
  Biologically inspired spiking neural networks (SNNs) are widely used to
realize ultralow-power energy consumption. However, deep SNNs are not easy to
train due to the excessive firing of spiking neurons in the hidden layers. To
tackle this problem, we propose a novel but simple normalization technique
called postsynaptic potential normalization. This normalization removes the
subtraction term from the standard normalization and uses the second raw moment
instead of the variance as the division term. The spike firing can be
controlled, enabling the training to proceed appropriating, by conducting this
simple normalization to the postsynaptic potential. The experimental results
show that SNNs with our normalization outperformed other models using other
normalizations. Furthermore, through the pre-activation residual blocks, the
proposed model can train with more than 100 layers without other special
techniques dedicated to SNNs.
