Be More Active! Understanding the Differences between Mean and Sampled
  Representations of Variational Autoencoders
  The ability of Variational Autoencoders to learn disentangled representations
has made them appealing for practical applications. However, their mean
representations, which are generally used for downstream tasks, have recently
been shown to be more correlated than their sampled counterpart, on which
disentanglement is usually measured. In this paper, we refine this observation
through the lens of selective posterior collapse, which states that only a
subset of the learned representations, the active variables, is encoding useful
information while the rest (the passive variables) is discarded. We first
extend the existing definition, originally proposed for sampled
representations, to mean representations and show that active variables are
equally disentangled in both representations. Based on this new definition and
the pre-trained models from disentanglement lib, we then isolate the passive
variables and show that they are responsible for the discrepancies between mean
and sampled representations. Specifically, passive variables exhibit high
correlation scores with other variables in mean representations while being
fully uncorrelated in sampled ones. We thus conclude that despite what their
higher correlation might suggest, mean representations are still good
candidates for downstream tasks applications. However, it may be beneficial to
remove their passive variables, especially when used with models sensitive to
correlated features.
