title: Is there any sense to use autoencoder for network with batch normalization?
body: <p>As known, main problem in DNN is long time of learning. </p>
  
  <p>But there are some ways to accelerate learning:</p>
  
  <ol>
  <li><strong>Batch Normalization</strong> <code>=(x-AVG)/Variance</code>: <a href="https://arxiv.org/abs/1502.03167" rel="nofollow noreferrer">https://arxiv.org/abs/1502.03167</a></li>
  </ol>
  
  <blockquote>
    <p>Batch Normalization achieves the same accuracy with <strong>14 times fewer
    training steps</strong></p>
  </blockquote>
  
  <ol start="2">
  <li><strong>ReLU</strong> <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)" rel="nofollow noreferrer"><code>=max(x, 0)</code></a> - rectified linear unit (ReLU,LReLU,PReLU,RReLU): <a href="https://arxiv.org/abs/1505.00853" rel="nofollow noreferrer">https://arxiv.org/abs/1505.00853</a></li>
  </ol>
  
  <blockquote>
    <p>The advantage of using non-saturated activation function lies in two
    aspects: The first is to solve the so called “exploding/vanishing
    gradient”. The second is to <strong>accelerate the convergence speed</strong>.</p>
  </blockquote>
  
  <p>Or any one: (maxout, ReLU-family, tanh)</p>
  
  <ol start="3">
  <li><strong>Fast weight initialization</strong> (with avoiding vanishing or exploding gradients): <a href="https://arxiv.org/abs/1511.06856" rel="nofollow noreferrer">https://arxiv.org/abs/1511.06856</a></li>
  </ol>
  
  <blockquote>
    <p>Our <strong>initialization</strong> matches the current state-of-the-art unsupervised
    or self-supervised pre-training methods on standard computer vision
    tasks, such as image classification and object detection, while being
    roughly <strong>three orders of magnitude faster</strong>.</p>
  </blockquote>
  
  <p>Or LSUV-initialization (Layer-sequential unit-variance): <a href="https://arxiv.org/abs/1511.06422" rel="nofollow noreferrer">https://arxiv.org/abs/1511.06422</a></p>
  
  <p>But if we use all steps: (1) Batch Normalization, (2) ReLU, (3) Fast weight initialization or LSUV - then is there any sense to use <a href="https://en.wikipedia.org/wiki/Autoencoder" rel="nofollow noreferrer">autoencoder/autoassociator</a> in any steps of training deep neural network?</p>