title: How to minimize the loss?
body: <p>This should be a regression problem.</p>
  
  <p>I would like the Neural Network to be able to estimate the length of a line, in pixel, from an image, like this 3 images, each image is 200 x 200 pcs:</p>
  
  <p>a)<a href="https://i.stack.imgur.com/5hAEG.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/5hAEG.jpg" alt="Train_1"></a>b)<a href="https://i.stack.imgur.com/uODWY.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uODWY.jpg" alt="Train_2"></a>c)<a href="https://i.stack.imgur.com/zbVK1.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/zbVK1.jpg" alt="Train_3"></a></p>
  
  <p>Training image of 6000 pcs, and validation image of 1000 pcs.</p>
  
  <p>The labels are the distance in pixel:</p>
  
  <p>a) 1.205404496424333018e+02</p>
  
  <p>b) 1.188780888137086436e+02</p>
  
  <p>c) 1.110180165558725918e+02</p>
  
  <p>Here is my training code:</p>
  
  <pre><code>img_size = 200
  
  def preprocess_image(image):
    image = tf.image.decode_jpeg(image, channels=3)
    image = tf.image.resize(image, [img_size, img_size])
    image /= 255.0  # normalize to [0,1] range
  
    return image
  
  def load_and_preprocess_image(path):
    image = tf.read_file(path)
    return preprocess_image(image)
  
  AUTOTUNE = tf.data.experimental.AUTOTUNE
  BATCH_SIZE = 16
  
  train_labels = np.loadtxt("train_labels.txt")
  val_labels = np.loadtxt("test_labels.txt")
  
  train_images = sorted(glob.glob("train_img/img_*.jpg"))
  val_images = sorted(glob.glob("test_img/img_*.jpg"))
  
  steps_per_epoch_count=tf.ceil(len(train_images)/BATCH_SIZE)
  
  train_path_ds = tf.data.Dataset.from_tensor_slices(train_images)
  val_path_ds = tf.data.Dataset.from_tensor_slices(val_images)
  
  train_image_ds = train_path_ds.map(load_and_preprocess_image, 
  num_parallel_calls = AUTOTUNE)
  train_label_ds = 
  tf.data.Dataset.from_tensor_slices(tf.cast(train_labels, tf.float32))
  train_image_label_ds = tf.data.Dataset.zip((train_image_ds, 
  train_label_ds))
  
  val_image_ds = val_path_ds.map(load_and_preprocess_image, 
  num_parallel_calls = AUTOTUNE)
  val_label_ds = tf.data.Dataset.from_tensor_slices(tf.cast(val_labels, tf.float32))
  val_image_label_ds = tf.data.Dataset.zip((val_image_ds, val_label_ds))
  
  
  model = tf.keras.models.Sequential([
      tf.keras.layers.Convolution2D(16,3,3, input_shape=(img_size, 
  img_size, 3), activation = 'relu'),
  
  tf.keras.layers.MaxPooling2D(pool_size=(2,2)),
  tf.keras.layers.Convolution2D(32,3,3, activation = 'relu'),
  tf.keras.layers.MaxPooling2D(pool_size=(2,2)),
  # tf.keras.layers.Convolution2D(64,3,3, activation = 'relu'),
  # tf.keras.layers.MaxPooling2D(pool_size=(2,2)),
  tf.keras.layers.Flatten(),
  tf.keras.layers.Dense(400, activation=tf.nn.relu),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(200, activation=tf.nn.relu),
  tf.keras.layers.Dropout(0.1),
  tf.keras.layers.Dense(100, activation=tf.nn.relu),
  tf.keras.layers.Dropout(0.05),
  tf.keras.layers.Dense(1, activation=tf.nn.relu)
  ])
  
  model.compile(optimizer=tf.keras.optimizers.RMSprop(0.01),
      loss = "mean_squared_error",
      metrics = ["mean_absolute_error", "mean_squared_error"]
      )
  
  train_ds = train_image_label_ds.apply(tf.data.experimental.shuffle_and_repeat(buffer_size=len(train_images)))
  train_ds = train_ds.batch(BATCH_SIZE)
  train_ds = train_ds.prefetch(buffer_size=AUTOTUNE)
  
  
  val_ds = val_image_label_ds.apply(
    tf.data.experimental.shuffle_and_repeat(buffer_size=len(val_images)))
  val_ds = val_ds.batch(BATCH_SIZE)
  val_ds = val_ds.prefetch(buffer_size=AUTOTUNE)
  
  history = model.fit(
      train_ds, 
      epochs = 80, 
      validation_data = val_ds,
      steps_per_epoch = 374,
      validation_steps = 62
      )
  </code></pre>
  
  <p>However, this is the train vs eval mean_squared_error plot:
  <a href="https://i.stack.imgur.com/NmGQw.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/NmGQw.png" alt="Plot"></a></p>
  
  <p>Question:</p>
  
  <ol>
  <li>Why is the validation loss not stable?</li>
  <li>The average Mean Squared Error is about 400 in training, which seems too high. What modification I can do to improve the estimation?</li>
  </ol>
  
  <p>EDIT:</p>
  
  <p>This is my latest model:</p>
  
  <p>Learning rate = 0.01</p>
  
  <p>Batch size = 16</p>
  
  <pre><code>model = tf.keras.models.Sequential([
      tf.keras.layers.Convolution2D(16,3,3, input_shape=(img_size, img_size, 3), activation = 'relu'),
      tf.keras.layers.MaxPooling2D(pool_size=(2,2)),
      tf.keras.layers.Convolution2D(32,3,3, activation = 'relu'),
      tf.keras.layers.MaxPooling2D(pool_size=(2,2)),
      tf.keras.layers.Flatten(),
      tf.keras.layers.Dense(2, activation=tf.nn.relu),
      tf.keras.layers.Dropout(0.5),
      tf.keras.layers.Dense(2, activation=tf.nn.relu), #, kernel_regularizer = tf.keras.regularizers.l2(0.001)
      tf.keras.layers.Dropout(0.5),
      tf.keras.layers.Dense(2, activation=tf.nn.relu), #, kernel_regularizer = tf.keras.regularizers.l2(0.001)
      tf.keras.layers.Dropout(0.5),
      tf.keras.layers.Dense(2, activation=tf.nn.relu), #, kernel_regularizer = tf.keras.regularizers.l2(0.001)
      tf.keras.layers.Dense(1, activation="linear")
  ])
  </code></pre>
  
  <p>The output looks like this:
  <a href="https://i.stack.imgur.com/mlWUR.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/mlWUR.png" alt="Very closed train and validation loss"></a></p>
  
  <p>As you can see, the train and validation loss is almost identical. The mse loss are both stabilized around 2393, which square root to 48.91 pixel error, quite high. </p>
  
  <p>What advice to lower it further? Is it normal?</p>