Prediction-Constrained Training for Semi-Supervised Mixture and Topic
  Models
  Supervisory signals have the potential to make low-dimensional data
representations, like those learned by mixture and topic models, more
interpretable and useful. We propose a framework for training latent variable
models that explicitly balances two goals: recovery of faithful generative
explanations of high-dimensional data, and accurate prediction of associated
semantic labels. Existing approaches fail to achieve these goals due to an
incomplete treatment of a fundamental asymmetry: the intended application is
always predicting labels from data, not data from labels. Our
prediction-constrained objective for training generative models coherently
integrates loss-based supervisory signals while enabling effective
semi-supervised learning from partially labeled data. We derive learning
algorithms for semi-supervised mixture and topic models using stochastic
gradient descent with automatic differentiation. We demonstrate improved
prediction quality compared to several previous supervised topic models,
achieving predictions competitive with high-dimensional logistic regression on
text sentiment analysis and electronic health records tasks while
simultaneously learning interpretable topics.
