Counterfactuals uncover the modular structure of deep generative models
  Deep generative models can emulate the perceptual properties of complex image
datasets, providing a latent representation of the data. However, manipulating
such representation to perform meaningful and controllable transformations in
the data space remains challenging without some form of supervision. While
previous work has focused on exploiting statistical independence to disentangle
latent factors, we argue that such requirement is too restrictive and propose
instead a non-statistical framework that relies on counterfactual manipulations
to uncover a modular structure of the network composed of disentangled groups
of internal variables. Experiments with a variety of generative models trained
on complex image datasets show the obtained modules can be used to design
targeted interventions. This opens the way to applications such as
computationally efficient style transfer and the automated assessment of
robustness to contextual changes in pattern recognition systems.
