title: Keras: How is Accuracy Calculated for Multi-Label Classification?
body: <p>I'm doing the Toxic Comment Text Classification Kaggle challenge.  There are 6 classes: <code>['threat', 'severe_toxic', 'obscene', 'insult', 'identity_hate', 'toxic']</code>.  A comment can be multiple of these classes so it's a multi-label classification problem.</p>
  
  <p>I built a basic neural network with Keras as follows:</p>
  
  <pre><code>model = Sequential()
  model.add(Embedding(10000, 128, input_length=250))
  model.add(Flatten())
  model.add(Dense(100, activation='relu'))
  model.add(Dense(len(classes), activation='sigmoid'))
  model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
  </code></pre>
  
  <p>I run this line:</p>
  
  <pre><code>model.fit(X_train, train_y, validation_split=0.5, epochs=3)
  </code></pre>
  
  <p>and get 99.11% accuracy after 3 epochs.</p>
  
  <p>However, 99.11% accuracy is a good bit higher than the best Kaggle submission.  This makes me think I'm either (possibly both) a) overfitting or b) misusing Keras's accuracy.</p>
  
  <p>1) Seems a bit hard to overfit when I'm using 50% of my data as a validation split and only 3 epochs.</p>
  
  <p>2) Is accuracy here just the percentage of the time the model gets each class correct?</p>
  
  <p>So if I output <code>[0, 0, 0, 0, 0, 1]</code> and the correct output was <code>[0, 0, 0, 0, 0, 0]</code>, my accuracy would be <code>5/6</code>?</p>
  
  <p>After a bit of thought, I sort of think the <code>accuracy</code> metric here is just looking at the class my model predicts with highest confidence and comparing vs. ground truth. </p>
  
  <p>So if my model outputs <code>[0, 0, 0.9, 0, 0, 0]</code>, it will compare the class at index 2 ('obscene') with the true value.  Do you think this is what's happening?</p>
  
  <p>Thanks for any help you can offer!</p>