title: Test accuracy always high regardless of how small my training set is
body: <p>I am doing a project where I am trying to classify comments into various categories: "toxic","severe_toxic","obscene","insult","identity_hate". The dataset I am using is from this Kaggle challenge: <a href="https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge" rel="nofollow noreferrer">https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge</a>. The current issue I am facing is that no matter how small a training dataset I fit my data on, when I predict labels for the test data, my accuracy is always around or above 90%. In this case I am training on 15 rows of data and testing on 159,556 rows. I would normally be excited to have a high testing accuracy, but in this case, I feel like I am doing something wrong.</p>
  
  <p>I am reading the data into a pandas dataframe:</p>
  
  <p><code>trainData = pd.read_csv('train.csv')</code></p>
  
  <p>Here is what the data looks like when printed:</p>
  
  <pre><code>                      id                                       comment_text  \
  0       0000997932d777bf  Explanation\nWhy the edits made under my usern...   
  1       000103f0d9cfb60f  D'aww! He matches this background colour I'm s...   
  2       000113f07ec002fd  Hey man, I'm really not trying to edit war. It...   
  3       0001b41b1c6bb37e  "\nMore\nI can't make any real suggestions on ...   
  4       0001d958c54c6e35  You, sir, are my hero. Any chance you remember...   
  ...                  ...                                                ...   
  159566  ffe987279560d7ff  ":::::And for the second time of asking, when ...   
  159567  ffea4adeee384e90  You should be ashamed of yourself \n\nThat is ...   
  159568  ffee36eab5c267c9  Spitzer \n\nUmm, theres no actual article for ...   
  159569  fff125370e4aaaf3  And it looks like it was actually you who put ...   
  159570  fff46fc426af1f9a  "\nAnd ... I really don't think you understand...   
  
          toxic  severe_toxic  obscene  threat  insult  identity_hate  
  0           0             0        0       0       0              0  
  1           0             0        0       0       0              0  
  2           0             0        0       0       0              0  
  3           0             0        0       0       0              0  
  4           0             0        0       0       0              0  
  ...       ...           ...      ...     ...     ...            ...  
  159566      0             0        0       0       0              0  
  159567      0             0        0       0       0              0  
  159568      0             0        0       0       0              0  
  159569      0             0        0       0       0              0  
  159570      0             0        0       0       0              0  
  
  [159571 rows x 8 columns]
  </code></pre>
  
  <p>Then I split the data into train and test, using train_test_split:</p>
  
  <pre><code>X = trainData.drop(labels= ['id','toxic','severe_toxic','obscene','threat','insult','identity_hate'],axis=1)
  Y = trainData.drop(labels = ['id','comment_text'],axis=1)
  
  trainX,testX,trainY,testY = train_test_split(X,Y,test_size=0.9999,random_state=99)
  </code></pre>
  
  <p>I am using sklearn's HashingVectorizer to convert the comments into numerical vectors for classifying:</p>
  
  <pre><code>def hashVec():
      trainComments=[]
      testComments=[]
      for index,row in trainX.iterrows():
          trainComments.append(row['comment_text'])
      for index,row in testX.iterrows():
          testComments.append(row['comment_text'])
      vectorizer = HashingVectorizer()
      trainSamples = vectorizer.transform(trainComments)
      testSamples = vectorizer.transform(testComments)
      return trainSamples,testSamples
  </code></pre>
  
  <p>I am using OneVsRestClassifier and LogisticRegression from sklearn to fit and predict data for each of the 6 classes</p>
  
  <pre><code>def logRegOVR(trainSamples,testSamples):
      commentTypes=['toxic','severe_toxic','obscene','threat','insult','identity_hate']
      clf = OneVsRestClassifier(LogisticRegression(solver='sag'))
      for cType in commentTypes:
          print(cType,":")
          clf.fit(trainSamples,trainY[cType])
          pred1 = clf.predict(trainSamples)
          print("\tTrain Accuracy:",accuracy_score(trainY[cType],pred1))
          prediction = clf.predict(testSamples)
          print("\tTest Accuracy:",accuracy_score(testY[cType],prediction))
  </code></pre>
  
  <p>Finally, here is where I call the functions, and the output I get:</p>
  
  <pre><code>sol = hashVec()
  logRegOVR(sol[0],sol[1])
  </code></pre>
  
  <pre><code>toxic :
      Train Accuracy: 0.8666666666666667
      Test Accuracy: 0.9041590413397177
  severe_toxic :
      Train Accuracy: 1.0
      Test Accuracy: 0.9900035097395272
  obscene :
      Train Accuracy: 1.0
      Test Accuracy: 0.9470468048835519
  threat :
      Train Accuracy: 1.0
      Test Accuracy: 0.9970041866178646
  insult :
      Train Accuracy: 1.0
      Test Accuracy: 0.9506317531148938
  identity_hate :
      Train Accuracy: 1.0
      Test Accuracy: 0.9911943142219659
  </code></pre>
  
  <p>The testing accuracy is very similar when I have a more reasonable train_test_split of 80% training and 20% testing.</p>
  
  <p>Thank you for the assistance</p>