On denoising autoencoders trained to minimise binary cross-entropy
  Denoising autoencoders (DAEs) are powerful deep learning models used for
feature extraction, data generation and network pre-training. DAEs consist of
an encoder and decoder which may be trained simultaneously to minimise a loss
(function) between an input and the reconstruction of a corrupted version of
the input. There are two common loss functions used for training autoencoders,
these include the mean-squared error (MSE) and the binary cross-entropy (BCE).
When training autoencoders on image data a natural choice of loss function is
BCE, since pixel values may be normalised to take values in [0,1] and the
decoder model may be designed to generate samples that take values in (0,1). We
show theoretically that DAEs trained to minimise BCE may be used to take
gradient steps in the data space towards regions of high probability under the
data-generating distribution. Previously this had only been shown for DAEs
trained using MSE. As a consequence of the theory, iterative application of a
trained DAE moves a data sample from regions of low probability to regions of
higher probability under the data-generating distribution. Firstly, we validate
the theory by showing that novel data samples, consistent with the training
data, may be synthesised when the initial data samples are random noise.
Secondly, we motivate the theory by showing that initial data samples
synthesised via other methods may be improved via iterative application of a
trained DAE to those initial samples.
