title: Simple NN model for next word prediction
body: <p>I tried to made a simple model of Neural Networks model to make next word prediction.</p>
  
  <p>I have a list of phrases and I split this list of phrases in a list of 3 words. E.g.</p>
  
  <p>Input: </p>
  
  <pre><code>1. This is phrase number one.
  2. This is phrase number two.
  </code></pre>
  
  <p>Result:</p>
  
  <pre><code>1. this is phrase
  2. is phrase number
  3. phrase number one
  ...
  n. phrase number two
  </code></pre>
  
  <p>And this is for me X values. For Y values I put the 4-th word from every phrase.
  E.g.</p>
  
  <pre><code>X[1]: this is phrase
  y[1]: number
  ...
  X[n]: is phrase number
  y[n]: two
  </code></pre>
  
  <p>And, in finally a encode the X and Y with value of 0 and 1. E.g.</p>
  
  <pre><code>X[0]: 0 0 0 ... 1 0 1 0 ... 1 - maximum 3 of '1'
  y[0]: 0 0 0 ... 1 0 0 0 ... 0 - maximum 1 of '1'
  </code></pre>
  
  <p>Where 1 in every vectors represent position of the word in my vocabulary.</p>
  
  <p>And that is my data. I'm not very sure about if this is the best way to embedding. Exist another way more better?</p>
  
  <p>Next, my model is like this:</p>
  
  <pre><code>model = Sequential()
  model.add(Dense(12, input_dim=len(X_train[0]), kernel_initializer='uniform', activation='relu'))
  model.add(Dense(8, input_dim=len(X_train[0]), kernel_initializer='uniform', activation ='relu'))
  model.add(Dense(len(X_train[0]), input_dim=len(X_train[0]), kernel_initializer='uniform', activation ='sigmoid'))
  
  # Compile model
  model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
  
  # Fit the model
  model.fit(X_train, y_train, epochs=10, batch_size=10, verbose=1)
  </code></pre>
  
  <p>But when I want to make a prediction I recived every time the same next word.</p>
  
  <p>Recover code for next word prediction:</p>
  
  <pre><code>    y_pred = model.predict(X_test)
      for i in range(0, len(X_test)):
      current_sentence = []
      pred_word = []
      for j in range(0, len(X_test[i])):
          if X_test[i][j] == 1:
              current_sentence.append(vocabulary[j])
  
      pred_word.append(vocabulary[list(y_pred[i]).index(max(y_pred[i]))])
  
      print("---------------\n")
      print("Current sentence: ", current_sentence)
      print("Next word prediction: ", pred_word)
  </code></pre>