title: Why are some neural network structures unable to fit a problem?
body: <p>I was playing around with an neural network code I wrote for <code>xor</code>-ing. I wrote this model:</p>
  <pre><code>import tensorflow as tf
  from tensorflow import keras
  import matplotlib.pyplot as plt
  
  model = keras.Sequential()
  model.add(tf.keras.Input(shape=(2,)))
  #model.add(keras.layers.Dense(2))
  model.add(keras.layers.Dense(4, activation='relu'))
  model.add(keras.layers.Dense(16, activation='relu'))
  model.add(keras.layers.Dense(1, activation='sigmoid'))
  
  optimizer = keras.optimizers.Adam(lr=0.2) #for example
  model.compile(optimizer=optimizer,
                loss=tf.losses.BinaryCrossentropy(),
                metrics=['accuracy'])
  
  # X_train, Y_train = ([[0, 0], [0, 1], [1, 0], [1, 1]], [[0], [1], [1], [0]])
  X_train = tf.cast([[0, 0], [0, 1], [1, 0], [1, 1]], tf.float32)
  Y_train = tf.cast([0, 1, 1, 0], tf.float32)
  
  # train model and plot loss
  hist = model.fit(X_train, Y_train, epochs=10000)
  plt.plot(hist.history['loss'])
  plt.show()
  
  print(model.predict([[0, 1]]))
  print(model.predict([[1, 1]]))
  print(model.predict([[1, 0]]))
  print(model.predict([[0, 0]]))
  </code></pre>
  <p>Half of the time I will get 100% accuracy in this model and half of the time I'll get only 75% (sometimes even 50%). Even after 10000 epochs the model can't find a way to fit!</p>
  <p>I've noticed that simply switching the layers as shown in the code below would yield 100% accuracy <strong>always</strong>.</p>
  <pre><code>model.add(tf.keras.Input(shape=(2,)))
  #model.add(keras.layers.Dense(2))
  #model.add(keras.layers.Dense(4, activation='relu'))
  model.add(keras.layers.Dense(16, activation='relu'))
  model.add(keras.layers.Dense(1, activation='sigmoid'))
  </code></pre>
  <h2>The Question</h2>
  <p>Why is this happening? Why can't this simple model fit well even after 10K epochs while the new model needs less than 500 epochs? What makes specifically the old model converge to that percentage?</p>