title: Understanding Huggingface BERT accuracy score in learner. Do all labels need to be correctly predicted to get counted as correct?
body: <p>I am working on a project using a Huggingface BERT implementation with the KTrain library in Python. I am trying to implement a binary-classifier with 6 labels (german political parties).
  While learning BERT returns the loss and accuracy value for each epoch. I noticed that all values are extremely low 10-20% accuracy. On the other hand, if I calculate the accuracy with sk-learn's accuracy_score I get accuracy values of 75-80% (Calculating the accuracy per label and afterwards taking the average of all 6 values).</p>
  <p>My guess is that the library calculates the accuracy score for all 6 labels simultaneously (e.g. all six labels need to be predicted correctly to count as correct). Reading the documentation about the library and BERT I couldn't determine how the accuracy is calculated. Does anybody know or have more information?</p>
  <p>My data:
  <a href="https://i.stack.imgur.com/C1qAd.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/C1qAd.png" alt="enter image description here" /></a></p>
  <p>My code:</p>
  <pre><code>#model_name = &quot;dbmdz/bert-base-german-cased&quot;
  model_name = &quot;deepset/gbert-base&quot;
  t = ktrain.text.Transformer(model_name, maxlen=64, class_names = parties)
  
  trn = t.preprocess_train(df_train[&quot;text&quot;].to_list(), df_train[parties].to_numpy())
  val = t.preprocess_test(df_val[&quot;text&quot;].to_list(), df_val[parties].to_numpy())
  
  model = t.get_classifier()
  learner = ktrain.get_learner(model, train_data=trn, val_data=val, batch_size=64)
  
  history_model = learner.autofit(1e-5, epochs=10)
  </code></pre>
  <blockquote>
  <p>begin training using triangular learning rate policy with max lr of
  1e-05... Epoch 1/10 38/38 [==============================] - 93s
  2s/step - loss: 0.6947 - accuracy: 0.1353 - val_loss: 0.6873 -
  val_accuracy: 0.0965 Epoch 2/10 38/38 [==============================]</p>
  <ul>
  <li>72s 2s/step - loss: 0.6891 - accuracy: 0.1382 - val_loss: 0.6837 - val_accuracy: 0.1555 Epoch 3/10 38/38 [==============================]</li>
  <li>71s 2s/step - loss: 0.6850 - accuracy: 0.1378 - val_loss: 0.6777 - val_accuracy: 0.1378 Epoch 4/10 38/38 [==============================]</li>
  <li>72s 2s/step - loss: 0.6770 - accuracy: 0.1408 - val_loss: 0.6689 - val_accuracy: 0.1437 Epoch 5/10 38/38 [==============================]</li>
  <li>72s 2s/step - loss: 0.6680 - accuracy: 0.1386 - val_loss: 0.6585 - val_accuracy: 0.1555 Epoch 6/10 38/38 [==============================]</li>
  <li>72s 2s/step - loss: 0.6591 - accuracy: 0.1601 - val_loss: 0.6471 - val_accuracy: 0.1713 Epoch 7/10 38/38 [==============================]</li>
  <li>72s 2s/step - loss: 0.6429 - accuracy: 0.1673 - val_loss: 0.6340 - val_accuracy: 0.1575 Epoch 8/10 38/38 [==============================]</li>
  <li>72s 2s/step - loss: 0.6263 - accuracy: 0.1580 - val_loss: 0.6211 - val_accuracy: 0.1417 Epoch 9/10 38/38 [==============================]</li>
  <li>72s 2s/step - loss: 0.6018 - accuracy: 0.1618 - val_loss: 0.6008 - val_accuracy: 0.1634 Epoch 10/10 38/38
  [==============================] - 72s 2s/step - loss: 0.5703 -
  accuracy: 0.1551 - val_loss: 0.5822 - val_accuracy: 0.1693</li>
  </ul>
  </blockquote>
  <pre><code>predictor = ktrain.get_predictor(learner.model, preproc=t)
  
  from sklearn.metrics import accuracy_score
  
  predictions = predictor.predict(df_val.text.tolist(),return_proba=True) 
  
  df_pred = pd.DataFrame(predictions,columns=parties, index=df_val.index)
  
  for party in parties:
    df_compare = pd.DataFrame(df_val[party])
    df_compare[&quot;predictions&quot;] = df_val[party]
    df_compare = df_compare[df_compare[party] != 0.5]
    df_compare.dropna(inplace= True)
    df_compare[&quot;predictions&quot;] = df_compare[&quot;predictions&quot;].round()
    print(f&quot;{party} accuracy score is: {round(accuracy_score(df_compare[party], df_compare['predictions']), 3)}&quot;)
  </code></pre>
  <blockquote>
  <p>cdu/csu accuracy score is: 0.778</p>
  </blockquote>
  <blockquote>
  <p>spd accuracy score is: 0.802</p>
  </blockquote>
  <blockquote>
  <p>grüne accuracy score is: 0.762</p>
  </blockquote>
  <blockquote>
  <p>fdp accuracy score is: 0.745</p>
  </blockquote>
  <blockquote>
  <p>die linke accuracy score is: 0.794</p>
  </blockquote>
  <blockquote>
  <p>afd accuracy score is: 0.782</p>
  </blockquote>