Thresholding Classifiers to Maximize F1 Score
  This paper provides new insight into maximizing F1 scores in the context of
binary classification and also in the context of multilabel classification. The
harmonic mean of precision and recall, F1 score is widely used to measure the
success of a binary classifier when one class is rare. Micro average, macro
average, and per instance average F1 scores are used in multilabel
classification. For any classifier that produces a real-valued output, we
derive the relationship between the best achievable F1 score and the
decision-making threshold that achieves this optimum. As a special case, if the
classifier outputs are well-calibrated conditional probabilities, then the
optimal threshold is half the optimal F1 score. As another special case, if the
classifier is completely uninformative, then the optimal behavior is to
classify all examples as positive. Since the actual prevalence of positive
examples typically is low, this behavior can be considered undesirable. As a
case study, we discuss the results, which can be surprising, of applying this
procedure when predicting 26,853 labels for Medline documents.
