title: How to tokenise text to use as input for a keras neural network
body: <p>I'm trying to perform sentiment analysis over a dataset composed of labeled english text labeled with a number between (0,4).</p>
  
  <p>I've been following the tensorflow guide on this from here: <a href="https://www.tensorflow.org/tutorials/keras/basic_text_classification" rel="nofollow noreferrer">https://www.tensorflow.org/tutorials/keras/basic_text_classification</a>
  adapted to suit my multiclass classification problem.</p>
  
  <p>A sample of the dataset is here:</p>
  
  <pre><code>PhraseId,SentenceId,Phrase,Sentiment
  21071,942,irony,1
  63332,3205,Blue Crush ' swims away with the Sleeper Movie of the Summer award .,2
  142018,7705,in the third row of the IMAX cinema,2
  103601,5464,images of a violent battlefield action picture,2 .
  12235,523,an engrossing story,3
  77679,3994,should come with the warning `` For serious film buffs only !,2
  58875,2969,enjoyed it,3
  152071,8297,"A delicious , quirky movie with a terrific screenplay and fanciful direction by Michael Gondry .",4
  </code></pre>
  
  <p>Currently, my model performs very badly, with a constant accuracy of about 0.5, and this doesn't change across epochs. </p>
  
  <p>I know how to tune the model's hyperparameters and all of the tricks I can try there, but nothing seems to help. I'm convinced that I've made a mistake somewhere in processing the data, since this is my first time doing deep learning with textual data.</p>
  
  <p>My current preprocessing consists of:</p>
  
  <ul>
  <li>Removing the PhraseID and SentenceID columns from the dataset</li>
  <li>Removing punctuation and upper case letters</li>
  <li>Shuffling the order of the dataset</li>
  <li>Separating the data and labels into different dataframes</li>
  <li>One-hot encoding the labels </li>
  <li>Tokenizing the data using the Keras preprocessing Tokenizer</li>
  <li>Padding the sequences to the same length</li>
  </ul>
  
  <p>I think there's an issue in the tokenization stage, or maybe I just don't understand how the model takes the tokenized words as an input vector and can learn from it.</p>
  
  <p>My relevant tokenization code is: </p>
  
  <pre><code>    def tokenize_data(self, df, max_features=5000):
      self.logger.log(f'Tokenizing with {max_features} features')
      tokenizer = Tokenizer(num_words=max_features, split=' ')
      tokenizer.fit_on_texts(df.values)
      train_set = tokenizer.texts_to_sequences(df.values)
      if self.logger.verbose_f : self.logger.verbose(train_set[:10])
      return train_set
  
  def pad_sequences(self, data, maxlen=5000):
      result = keras.preprocessing.sequence.pad_sequences(data,
                                                      value=0,
                                                      padding='post',
                                                      maxlen=maxlen)
      if self.logger.verbose_f:
          df = pd.DataFrame(result)
          df.to_csv("processed.csv")
  
      return result
  </code></pre>
  
  <p>The output of the pad sequences looks like this:</p>
  
  <pre><code>7,821,3794,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
  
  8,74,44,344,325,2904,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
  </code></pre>
  
  <p>etc etc for each instance. </p>
  
  <p>These values get fed into the model like this to act as the training data.</p>
  
  <p>Do I need to do some sort of normalisation before I give train on this? </p>
  
  <p>Or am I completely barking up the wrong tree? </p>
  
  <p>Thanks</p>