Stochastic Bottleneck: Rateless Auto-Encoder for Flexible Dimensionality
  Reduction
  We propose a new concept of rateless auto-encoders (RL-AEs) that enable a
flexible latent dimensionality, which can be seamlessly adjusted for varying
distortion and dimensionality requirements. In the proposed RL-AEs, instead of
a deterministic bottleneck architecture, we use an over-complete representation
that is stochastically regularized with weighted dropouts, in a manner
analogous to sparse AE (SAE). Unlike SAEs, our RL-AEs employ monotonically
increasing dropout rates across the latent representation nodes such that the
latent variables become sorted by importance like in principal component
analysis (PCA). This is motivated by the rateless property of conventional PCA,
where the least important principal components can be discarded to realize
variable rate dimensionality reduction that gracefully degrades the distortion.
In contrast, since the latent variables of conventional AEs are equally
important for data reconstruction, they cannot be simply discarded to further
reduce the dimensionality after the AE model is trained. Our proposed
stochastic bottleneck framework enables seamless rate adaptation with high
reconstruction performance, without requiring predetermined latent
dimensionality at training. We experimentally demonstrate that the proposed
RL-AEs can achieve variable dimensionality reduction while achieving low
distortion compared to conventional AEs.
