Discovering Concepts in Learned Representations using Statistical
  Inference and Interactive Visualization
  Concept discovery is one of the open problems in the interpretability
literature that is important for bridging the gap between non-deep learning
experts and model end-users. Among current formulations, concepts defines them
by as a direction in a learned representation space. This definition makes it
possible to evaluate whether a particular concept significantly influences
classification decisions for classes of interest. However, finding relevant
concepts is tedious, as representation spaces are high-dimensional and hard to
navigate. Current approaches include hand-crafting concept datasets and then
converting them to latent space directions; alternatively, the process can be
automated by clustering the latent space. In this study, we offer another two
approaches to guide user discovery of meaningful concepts, one based on
multiple hypothesis testing, and another on interactive visualization. We
explore the potential value and limitations of these approaches through
simulation experiments and an demo visual interface to real data. Overall, we
find that these techniques offer a promising strategy for discovering relevant
concepts in settings where users do not have predefined descriptions of them,
but without completely automating the process.
