Do autoencoders need a bottleneck for anomaly detection?
  A common belief in designing deep autoencoders (AEs), a type of unsupervised
neural network, is that a bottleneck is required to prevent learning the
identity function. Learning the identity function renders the AEs useless for
anomaly detection. In this work, we challenge this limiting belief and
investigate the value of non-bottlenecked AEs.
  The bottleneck can be removed in two ways: (1) overparameterising the latent
layer, and (2) introducing skip connections. However, limited works have
reported on the use of one of the ways. For the first time, we carry out
extensive experiments covering various combinations of bottleneck removal
schemes, types of AEs and datasets. In addition, we propose the infinitely-wide
AEs as an extreme example of non-bottlenecked AEs.
  Their improvement over the baseline implies learning the identity function is
not trivial as previously assumed. Moreover, we find that non-bottlenecked
architectures (highest AUROC=0.857) can outperform their bottlenecked
counterparts (highest AUROC=0.696) on the popular task of CIFAR (inliers) vs
SVHN (anomalies), among other tasks, shedding light on the potential of
developing non-bottlenecked AEs for improving anomaly detection.
