title: Inference results depend on order of images in training batch
body: <p>I've trained the same network two times on the same dataset of 5 images. For the first time, the images in a batch for each step were in the same order. For the second time, the batch was shuffled before every training step. Both models overfit. Both models were tested on shuffled images from training dataset.
  The firs model shows 100% accuracy.
  ```</p>
  
  <pre><code>Prediction  Labels
  [6 1 4 3 7] [6 1 4 3 7]
  [3 4 1 7 6] [3 4 1 7 6]
  [6 1 4 3 7] [6 1 4 3 7]
  [4 3 7 6 1] [4 3 7 6 1]
  [4 7 6 3 1] [4 7 6 3 1]
  [1 3 7 6 4] [1 3 7 6 4]
  [3 1 6 7 4] [3 1 6 7 4]
  </code></pre>
  
  <p>```</p>
  
  <p>The second model shows accuracy close to a random guess.
  ```</p>
  
  <pre><code>Prediction  Labels
  [1 4 3 6 7] [7 4 6 3 1]
  [3 6 4 1 7] [7 4 3 1 6]
  [1 6 3 7 4] [1 6 4 3 7]
  [4 7 3 6 1] [1 6 4 3 7]
  [3 6 7 4 1] [4 1 3 7 6]
  [1 3 7 4 6] [6 3 4 7 1]
  [1 4 7 3 6] [3 7 1 4 6]
  </code></pre>
  
  <p>```</p>
  
  <p>Related issues - none</p>
  
  <h3>Environment:</h3>
  
  <ul>
  <li>Ubuntu 14.04 64-bit</li>
  <li>CUDA 8.0</li>
  <li>cuDNN 5.1</li>
  <li>tensorflow 0.12.1 from pip with GPU support</li>
  </ul>
  
  <h3>Dataset</h3>
  
  <p>5 images from MNIST</p>
  
  <h3>The attached archive</h3>
  
  <p>contains code for reproducible example. </p>
  
  <ul>
  <li>runner.py is responsible for preparing image batches and maintains the queue.</li>
  <li>network.py contains the code of simple neural network(2 convolutional layers with relu activation, softmax output layer)</li>
  <li>train_not_shuffled.py trains and tests the network using non-shuffled batch</li>
  <li>train_shuffled.py trains and tests the network using shuffled batch</li>
  </ul>
  
  <p><a href="https://github.com/tensorflow/tensorflow/files/735318/example.gz" rel="nofollow noreferrer">example.gz</a></p>
  
  <p>I've retrained the models several times, tried different datasets and network architectures. I've visualized graph to make sure inference uses the same variables. I've also visualized weights, they look slightly different for two cases but don't contain any anomalies that could explain such behavior.</p>