Sparsely Activated Networks: A new method for decomposing and
  compressing data
  Recent literature on unsupervised learning focused on designing structural
priors with the aim of learning meaningful features, but without considering
the description length of the representations. In this thesis, first we
introduce the $\varphi$ metric that evaluates unsupervised models based on
their reconstruction accuracy and the degree of compression of their internal
representations. We then present and define two activation functions (Identity,
ReLU) as base of reference and three sparse activation functions (top-k
absolutes, Extrema-Pool indices, Extrema) as candidate structures that minimize
the previously defined metric $\varphi$. We lastly present Sparsely Activated
Networks (SANs) that consist of kernels with shared weights that, during
encoding, are convolved with the input and then passed through a sparse
activation function. During decoding, the same weights are convolved with the
sparse activation map and subsequently the partial reconstructions from each
weight are summed to reconstruct the input. We compare SANs using the five
previously defined activation functions on a variety of datasets (Physionet,
UCI-epilepsy, MNIST, FMNIST) and show that models that are selected using
$\varphi$ have small description representation length and consist of
interpretable kernels.
