Auto-clustering Output Layer: Automatic Learning of Latent Annotations
  in Neural Networks
  In this paper, we discuss a different type of semi-supervised setting: a
coarse level of labeling is available for all observations but the model has to
learn a fine level of latent annotation for each one of them. Problems in this
setting are likely to be encountered in many domains such as text
categorization, protein function prediction, image classification as well as in
exploratory scientific studies such as medical and genomics research. We
consider this setting as simultaneously performed supervised classification
(per the available coarse labels) and unsupervised clustering (within each one
of the coarse labels) and propose a novel output layer modification called
auto-clustering output layer (ACOL) that allows concurrent classification and
clustering based on Graph-based Activity Regularization (GAR) technique. As the
proposed output layer modification duplicates the softmax nodes at the output
layer for each class, GAR allows for competitive learning between these
duplicates on a traditional error-correction learning framework to ultimately
enable a neural network to learn the latent annotations in this partially
supervised setup. We demonstrate how the coarse label supervision impacts
performance and helps propagate useful clustering information between
sub-classes. Comparative tests on three of the most popular image datasets
MNIST, SVHN and CIFAR-100 rigorously demonstrate the effectiveness and
competitiveness of the proposed approach.
