title: Siamese Network for binary classification with pre-encoded inputs
body: <p>I want to train a Siamese Network to compare vectors for similarity.</p>
  <p>My dataset consist of pairs of vectors and a target column with &quot;1&quot; if they are the same and &quot;0&quot; otherwise (binary classification):</p>
  <pre class="lang-py prettyprint-override"><code>import pandas as pd
  
  # Define train and test sets.
  X_train_val = pd.read_csv(&quot;train.csv&quot;)
  print(X_train_val.head())
  
  y_train_val = X_train_val.pop(&quot;class&quot;)
  print(y_train_val.value_counts())
  
  # Keep 50% of X_train_val in validation set.
  X_train, X_val = X_train_val[:991], X_train_val[991:]
  y_train, y_val = y_train_val[:991], y_train_val[991:]
  del X_train_val, y_train_val
  
  # Split our data to 'left' and 'right' inputs (one for each side Siamese).
  X_left_train, X_right_train = X_train.iloc[:, :200], X_train.iloc[:, 200:]
  X_left_val, X_right_val = X_val.iloc[:, :200], X_val.iloc[:, 200:]
  
  assert X_left_train.shape == X_right_train.shape
  
  # Repeat for test set.
  X_test = pd.read_csv(&quot;test.csv&quot;)
  y_test = X_test.pop(&quot;class&quot;)
  
  print(y_test.value_counts())
  
  X_left_test, X_right_test = X_test.iloc[:, :200], X_test.iloc[:, 200:]
  </code></pre>
  <p>returns</p>
  <pre><code>         v0        v1        v2  ...       v397      v398      v399  class
  0  0.003615  0.013794  0.030388  ...  -0.093931  0.106202  0.034870    0.0
  1  0.018988  0.056302  0.002915  ...  -0.007905  0.100859 -0.043529    0.0
  2  0.072516  0.125697  0.111230  ...  -0.010007  0.064125 -0.085632    0.0
  3  0.051016  0.066028  0.082519  ...   0.012677  0.043831 -0.073935    1.0
  4  0.020367  0.026446  0.015681  ...   0.062367 -0.022781 -0.032091    0.0
  
  1.0    1060
  0.0     923
  Name: class, dtype: int64
  
  1.0     354
  0.0     308
  Name: class, dtype: int64
  </code></pre>
  <p>The rest of my script is as follows:</p>
  <pre class="lang-py prettyprint-override"><code>import keras
  import keras.backend as K
  from keras.layers import Dense, Dropout, Input, Lambda
  from keras.models import Model
  
  
  def euclidean_distance(vectors):
      &quot;&quot;&quot;
      Find the Euclidean distance between two vectors.
      &quot;&quot;&quot;
      x, y = vectors
      sum_square = K.sum(K.square(x - y), axis=1, keepdims=True)
      # Epsilon is small value that makes very little difference to the value of the denominator, but ensures that it isn't equal to exactly zero.
      return K.sqrt(K.maximum(sum_square, K.epsilon()))
  
  
  def contrastive_loss(y_true, y_pred):
      &quot;&quot;&quot;
      Distance-based loss function that tries to ensure that data samples that are semantically similar are embedded closer together.
  
      See:
      * https://gombru.github.io/2019/04/03/ranking_loss/
      &quot;&quot;&quot;
      margin = 1
      return K.mean(y_true * K.square(y_pred) + (1 - y_true) * K.square(K.maximum(margin - y_pred, 0)))
  
  
  def accuracy(y_true, y_pred):
      &quot;&quot;&quot;
      Compute classification accuracy with a fixed threshold on distances.
      &quot;&quot;&quot;
      return K.mean(K.equal(y_true, K.cast(y_pred &lt; 0.5, y_true.dtype)))
  
  
  def create_base_network(input_dim: int, dense_units: int, dropout_rate: float):
      input1 = Input(input_dim, name=&quot;encoder&quot;)
      x = input1
      x = Dense(dense_units, activation=&quot;relu&quot;)(x)
      x = Dropout(dropout_rate)(x)
      x = Dense(dense_units, activation=&quot;relu&quot;)(x)
      x = Dropout(dropout_rate)(x)
      x = Dense(dense_units, activation=&quot;relu&quot;, name=&quot;Embeddings&quot;)(x)
      return Model(input1, x)
  
  
  def build_siamese_model(input_dim: int):
      shared_network = create_base_network(input_dim, dense_units=128, dropout_rate=0.1)
  
      left_input = Input(input_dim)
      right_input = Input(input_dim)
  
      # Since this is a siamese nn, both sides share the same network.
      encoded_l = shared_network(left_input)
      encoded_r = shared_network(right_input)
  
      # The euclidean distance layer outputs close to 0 value when two inputs are similar and 1 otherwise.
      distance = Lambda(euclidean_distance, name=&quot;Euclidean-Distance&quot;)([encoded_l, encoded_r])
  
      siamese_net = Model(inputs=[left_input, right_input], outputs=distance)
      siamese_net.compile(loss=contrastive_loss, optimizer=&quot;RMSprop&quot;, metrics=[accuracy])
  
      return siamese_net
  
  
  model = build_siamese_model(X_left_train.shape[1])
  
  es_callback = keras.callbacks.EarlyStopping(monitor=&quot;val_loss&quot;, patience=3, verbose=0)
  history = model.fit(
      [X_left_train, X_right_train],
      y_train,
      validation_data=([X_left_val, X_right_val], y_val),
      epochs=100,
      callbacks=[es_callback],
      verbose=1,
  )
  </code></pre>
  <p>I have plotted the contrastive loss vs epoch and model accuracy vs epoch:</p>
  <p><a href="https://i.stack.imgur.com/B11Lr.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/B11Lr.png" alt="history" /></a></p>
  <p>The validation line is almost flat, which seems odd to me (overfitted?).</p>
  <p>After changing the dropout of the shared network from 0.1 to 0.5, I get the following results:</p>
  <p><a href="https://i.stack.imgur.com/EfHnS.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/EfHnS.png" alt="history-2" /></a></p>
  <p>Somehow it looks better, but yields bad predictions as well.</p>
  <p>My questions are:</p>
  <ul>
  <li><p>Most examples of Siamese Networks I've seen so far involves embedding layers (text pairs) and/or Convolution layers (image pairs). My input pairs are the actual vector representation of some text, which is why I used <code>Dense</code> layers for the shared network. Is this the proper approach?</p>
  </li>
  <li><p>The output layer of my Siamese Network is as follows:</p>
  <pre class="lang-py prettyprint-override"><code>distance = Lambda(euclidean_distance, name=&quot;Euclidean-Distance&quot;)([encoded_l, encoded_r])
  siamese_net = Model(inputs=[left_input, right_input], outputs=distance)
  siamese_net.compile(loss=contrastive_loss, optimizer=&quot;RMSprop&quot;, metrics=[accuracy])
  </code></pre>
  <p>but <em>someone</em> over the internet suggested this instead:</p>
  <pre class="lang-py prettyprint-override"><code>distance = Lambda(lambda tensors: K.abs(tensors[0] - tensors[1]), name=&quot;L1-Distance&quot;)([encoded_l, encoded_r])
  output = Dense(1, activation=&quot;sigmoid&quot;)(distance)  # returns the class probability
  siamese_net = Model(inputs=[left_input, right_input], outputs=output)
  siamese_net.compile(loss=&quot;binary_crossentropy&quot;, optimizer=&quot;adam&quot;, metrics=[&quot;accuracy&quot;])
  </code></pre>
  <p>I'm not sure which one to trust nor the difference between them (except that the former returns the distance and the latter returns the class probability). In my experiments, I get poor results with <code>binary_crossentropy</code>.</p>
  </li>
  </ul>
  <p><strong>EDIT:</strong></p>
  <p>After following @PlzBePython suggestions, I come up with the following base network:</p>
  <pre class="lang-py prettyprint-override"><code>distance = Lambda(lambda tensors: K.abs(tensors[0] - tensors[1]), name=&quot;L1-Distance&quot;)([encoded_l, encoded_r])
  output = Dense(1, activation=&quot;linear&quot;)(distance)
  siamese_net = Model(inputs=[left_input, right_input], outputs=output)
  siamese_net.compile(loss=contrastive_loss, optimizer=&quot;RMSprop&quot;, metrics=[accuracy])
  </code></pre>
  <p><a href="https://i.stack.imgur.com/zV0Uf.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/zV0Uf.png" alt="history-3" /></a></p>
  <p>Thank you for your help!</p>