Mixture of Linear Models Co-supervised by Deep Neural Networks
  Deep neural network (DNN) models have achieved phenomenal success for
applications in many domains, ranging from academic research in science and
engineering to industry and business. The modeling power of DNN is believed to
have come from the complexity and over-parameterization of the model, which on
the other hand has been criticized for the lack of interpretation. Although
certainly not true for every application, in some applications, especially in
economics, social science, healthcare industry, and administrative decision
making, scientists or practitioners are resistant to use predictions made by a
black-box system for multiple reasons. One reason is that a major purpose of a
study can be to make discoveries based upon the prediction function, e.g., to
reveal the relationships between measurements. Another reason can be that the
training dataset is not large enough to make researchers feel completely sure
about a purely data-driven result. Being able to examine and interpret the
prediction function will enable researchers to connect the result with existing
knowledge or gain insights about new directions to explore. Although classic
statistical models are much more explainable, their accuracy often falls
considerably below DNN. In this paper, we propose an approach to fill the gap
between relatively simple explainable models and DNN such that we can more
flexibly tune the trade-off between interpretability and accuracy. Our main
idea is a mixture of discriminative models that is trained with the guidance
from a DNN. Although mixtures of discriminative models have been studied
before, our way of generating the mixture is quite different.
