Statistical Characteristics of Deep Representations: An Empirical
  Investigation
  In this study, the effects of eight representation regularization methods are
investigated, including two newly developed rank regularizers (RR). The
investigation shows that the statistical characteristics of representations
such as correlation, sparsity, and rank can be manipulated as intended, during
training. Furthermore, it is possible to improve the baseline performance
simply by trying all the representation regularizers and fine-tuning the
strength of their effects. In contrast to performance improvement, no
consistent relationship between performance and statistical characteristics was
observable. The results indicate that manipulation of statistical
characteristics can be helpful for improving performance, but only indirectly
through its influence on learning dynamics or its tuning effects.
