Representation learnt by SGD and Adaptive learning rules -- Conditions
  that Vary Sparsity and Selectivity in Neural Network
  From the point of view of the human brain, continual learning can perform
various tasks without mutual interference. An effective way to reduce mutual
interference can be found in sparsity and selectivity of neurons. According to
Aljundi et al. and Hadsell et al., imposing sparsity at the representational
level is advantageous for continual learning because sparse neuronal
activations encourage less overlap between parameters, resulting in less
interference. Similarly, highly selective neural networks are likely to induce
less interference since particular response in neurons will reduce the chance
of overlap with other parameters. Considering that the human brain performs
continual learning over the lifespan, finding conditions where sparsity and
selectivity naturally arises may provide insight for understanding how the
brain functions. This paper investigates various conditions that naturally
increase sparsity and selectivity in a neural network. This paper tested
different optimizers with Hoyer's sparsity metric and CCMAS selectivity metric
in MNIST classification task. It is essential to note that investigations on
the natural occurrence of sparsity and selectivity concerning various
conditions have not been acknowledged in any sector of neuroscience nor machine
learning until this day. This paper found that particular conditions increase
sparsity and selectivity such as applying a large learning rate and lowering a
batch size. In addition to the relationship between the condition, sparsity,
and selectivity, the following will be discussed based on empirical analysis:
1. The relationship between sparsity and selectivity and 2. The relationship
between test accuracy, sparsity, and selectivity.
