title: Tensorflow Embedding layer triggering an InvalidArgumentError
body: <p>I am using a two tower neural network for a recommender system.
  The way i interpret the error log is that on row 32 col 0 of the train dataset, it finds a value of 329416640 that doesn't fit in the range of [0, 853515). I have absolutely no idea what this value of 329416640 corresponds to.</p>
  <p>I use the following dataset.</p>
  <p><a href="https://i.stack.imgur.com/603Gg.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/603Gg.png" alt="enter image description here" /></a></p>
  <p>the dataset has been split into a train and a test split of 80/20.
  Shape of train data: (4728919, 3)
  Shape of test data: (1182230, 3)</p>
  <p>Then i determine my vocab_lengths as follows:</p>
  <pre><code>client_vocab_len = (int(client_data.CLI_ID.nunique()))
  libelle_vocab_len = (int(items_data.prod_id.nunique()))
  print(client_vocab_len) # = 853514
  print(libelle_vocab_len) # = 1484
  
  </code></pre>
  <p>My model architecture looks like this:</p>
  <pre><code>#user input network
  input_clients = tf.layers.Input(shape=[1])
  dense = tf.layers.Dense(32, activation=&quot;relu&quot;)
  yy = dense(input_clients)
  embed_clients = tf.layers.Embedding(client_vocab_len+1, 15)(yy)
  clients_out = tf.layers.Flatten()(embed_clients)
  
  #product input network
  input_products = tf.layers.Input(shape=[1])
  dense = tf.layers.Dense(32, activation=&quot;relu&quot;)
  xx = dense(input_products)
  embed_products = tf.layers.Embedding(libelle_vocab_len+1, 15)(xx)
  products_out = tf.layers.Flatten()(embed_products)
  
  
  conc_layer = tf.layers.Concatenate()([clients_out, products_out])
  x = tf.layers.Dense(128, activation='relu')(conc_layer)
  #x = tf.layers.Dropout(0.3)(x)
  x = tf.layers.Dense(128, activation='relu')(x)
  x_out = x = tf.layers.Dense(1, activation='relu')(x)
  model = tf.Model([input_clients, input_products], x_out)
  </code></pre>
  <p><a href="https://i.stack.imgur.com/CfBON.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/CfBON.png" alt="enter image description here" /></a></p>
  <p>when i try to train my model, i get the following error log:</p>
  <pre><code># Create a callback to stop the model if overfits
  early_stop = EarlyStopping(monitor='val_loss',patience=3) 
  lr_reduce = ReduceLROnPlateau(monitor='val_loss',
                                factor=0.75,
                                patience=2,
                                verbose=2,
                                mode='auto')
  
  
  hist = model.fit([Xtrain.CLI_ID, Xtrain.prod_id], Xtrain.QTY, 
                   batch_size=128, 
                   epochs=15, 
                   verbose=1,
                   validation_data=([Xtest.CLI_ID, Xtest.prod_id], Xtest.QTY),
                   callbacks=[early_stop,lr_reduce,PlotLossesKeras()])
  
  
  
  ---------------------------------------------------------------------------
  InvalidArgumentError                      Traceback (most recent call last)
  &lt;ipython-input-166-d767717c6437&gt; in &lt;module&gt;
        8 
        9 
  ---&gt; 10 hist = model.fit([Xtrain.CLI_ID, Xtrain.prod_id], Xtrain.QTY, 
       11                  batch_size=128,
       12                  epochs=15,
  
  ~/opt/anaconda3/lib/python3.8/site-packages/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
     1182                 _r=1):
     1183               callbacks.on_train_batch_begin(step)
  -&gt; 1184               tmp_logs = self.train_function(iterator)
     1185               if data_handler.should_sync:
     1186                 context.async_wait()
  
  ~/opt/anaconda3/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
      883 
      884       with OptionalXlaContext(self._jit_compile):
  --&gt; 885         result = self._call(*args, **kwds)
      886 
      887       new_tracing_count = self.experimental_get_tracing_count()
  
  ~/opt/anaconda3/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
      948         # Lifting succeeded, so variables are initialized and we can run the
      949         # stateless function.
  --&gt; 950         return self._stateless_fn(*args, **kwds)
      951     else:
      952       _, _, _, filtered_flat_args = \
  
  ~/opt/anaconda3/lib/python3.8/site-packages/tensorflow/python/eager/function.py in __call__(self, *args, **kwargs)
     3037       (graph_function,
     3038        filtered_flat_args) = self._maybe_define_function(args, kwargs)
  -&gt; 3039     return graph_function._call_flat(
     3040         filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
     3041 
  
  ~/opt/anaconda3/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
     1961         and executing_eagerly):
     1962       # No tape is watching; skip to running the function.
  -&gt; 1963       return self._build_call_outputs(self._inference_function.call(
     1964           ctx, args, cancellation_manager=cancellation_manager))
     1965     forward_backward = self._select_forward_and_backward_functions(
  
  ~/opt/anaconda3/lib/python3.8/site-packages/tensorflow/python/eager/function.py in call(self, ctx, args, cancellation_manager)
      589       with _InterpolateFunctionError(self):
      590         if cancellation_manager is None:
  --&gt; 591           outputs = execute.execute(
      592               str(self.signature.name),
      593               num_outputs=self._num_outputs,
  
  ~/opt/anaconda3/lib/python3.8/site-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
       57   try:
       58     ctx.ensure_initialized()
  ---&gt; 59     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
       60                                         inputs, attrs, num_outputs)
       61   except core._NotOkStatusException as e:
  
  InvalidArgumentError:  indices[32,0] = 329416640 is not in [0, 853515)
       [[node model_27/embedding_55/embedding_lookup (defined at &lt;ipython-input-166-d767717c6437&gt;:10) ]] [Op:__inference_train_function_28184]
  
  Errors may have originated from an input operation.
  Input Source operations connected to node model_27/embedding_55/embedding_lookup:
   model_27/embedding_55/embedding_lookup/27907 (defined at /Users/jonathankhalifa/opt/anaconda3/lib/python3.8/contextlib.py:113)
  
  Function call stack:
  train_function
  
  
  </code></pre>