title: Intuition behind Stacking Multiple Conv2D Layers before Dropout in CNN
body: <p><strong>Background</strong>:</p>
  
  <p><em>Tagging TensorFlow since Keras runs on top of it and this is more a general deep learning question.</em></p>
  
  <p>I have been working on the Kaggle Digit Recognizer problem and used Keras to train CNN models for the task. This model below has the original CNN structure I used for this competition and it performed okay.</p>
  
  <pre><code>def build_model1():
      model = models.Sequential()
  
      model.add(layers.Conv2D(32, (3, 3), padding="Same" activation="relu", input_shape=[28, 28, 1]))
      model.add(layers.MaxPooling2D((2, 2)))
      model.add(layers.Dropout(0.25))
  
      model.add(layers.Conv2D(64, (3, 3), padding="Same", activation="relu"))
      model.add(layers.MaxPooling2D((2, 2)))
      model.add(layers.Dropout(0.25))
  
      model.add(layers.Conv2D(64, (3, 3), padding="Same", activation="relu"))
      model.add(layers.MaxPooling2D((2, 2)))
      model.add(layers.Dropout(0.25))
  
      model.add(layers.Flatten())
      model.add(layers.Dense(64, activation="relu"))
      model.add(layers.Dropout(0.5))
      model.add(layers.Dense(10, activation="softmax"))
  
      return model
  </code></pre>
  
  <p>Then I read some other notebooks on Kaggle and borrowed another CNN structure (copied below), which works much better than the one above in that it achieved better accuracy, lower error rate, and took many more epochs before overfitting the training data.</p>
  
  <pre><code>def build_model2():
      model = models.Sequential()
  
      model.add(layers.Conv2D(32, (5, 5),padding ='Same', activation='relu', input_shape = (28, 28, 1)))
      model.add(layers.Conv2D(32, (5, 5),padding = 'Same', activation ='relu'))
      model.add(layers.MaxPool2D((2, 2)))
      model.add(layers.Dropout(0.25))
  
      model.add(layers.Conv2D(64,(3, 3),padding = 'Same', activation ='relu'))
      model.add(layers.Conv2D(64, (3, 3),padding = 'Same', activation ='relu'))
      model.add(layers.MaxPool2D(pool_size=(2, 2), strides=(2, 2)))
      model.add(layers.Dropout(0.25))
  
      model.add(layers.Flatten())
      model.add(layers.Dense(256, activation = "relu"))
      model.add(layers.Dropout(0.5))
      model.add(layers.Dense(10, activation = "softmax"))
  
      return model
  </code></pre>
  
  <p><strong>Question</strong>: </p>
  
  <p>Is there any intuition or explanation behind the better performance of the second CNN structure? What is it that makes stacking 2 Conv2D layers better than just using 1 Conv2D layer before max pooling and dropout? Or is there something else that contributes to the result of the second model?</p>
  
  <p>Thank y'all for your time and help.</p>