Pretraining Image Encoders without Reconstruction via Feature Prediction
  Loss
  This work investigates three methods for calculating loss for
autoencoder-based pretraining of image encoders: The commonly used
reconstruction loss, the more recently introduced deep perceptual similarity
loss, and a feature prediction loss proposed here; the latter turning out to be
the most efficient choice. Standard auto-encoder pretraining for deep learning
tasks is done by comparing the input image and the reconstructed image. Recent
work shows that predictions based on embeddings generated by image autoencoders
can be improved by training with perceptual loss, i.e., by adding a loss
network after the decoding step. So far the autoencoders trained with loss
networks implemented an explicit comparison of the original and reconstructed
images using the loss network. However, given such a loss network we show that
there is no need for the time-consuming task of decoding the entire image.
Instead, we propose to decode the features of the loss network, hence the name
"feature prediction loss". To evaluate this method we perform experiments on
three standard publicly available datasets (LunarLander-v2, STL-10, and SVHN)
and compare six different procedures for training image encoders (pixel-wise,
perceptual similarity, and feature prediction losses; combined with two
variations of image and feature encoding/decoding). The embedding-based
prediction results show that encoders trained with feature prediction loss is
as good or better than those trained with the other two losses. Additionally,
the encoder is significantly faster to train using feature prediction loss in
comparison to the other losses. The method implementation used in this work is
available online: https://github.com/guspih/Perceptual-Autoencoders
