Deep F-measure Maximization for End-to-End Speech Understanding
  Spoken language understanding (SLU) datasets, like many other machine
learning datasets, usually suffer from the label imbalance problem. Label
imbalance usually causes the learned model to replicate similar biases at the
output which raises the issue of unfairness to the minority classes in the
dataset. In this work, we approach the fairness problem by maximizing the
F-measure instead of accuracy in neural network model training. We propose a
differentiable approximation to the F-measure and train the network with this
objective using standard backpropagation. We perform experiments on two
standard fairness datasets, Adult, and Communities and Crime, and also on
speech-to-intent detection on the ATIS dataset and speech-to-image concept
classification on the Speech-COCO dataset. In all four of these tasks,
F-measure maximization results in improved micro-F1 scores, with absolute
improvements of up to 8% absolute, as compared to models trained with the
cross-entropy loss function. In the two multi-class SLU tasks, the proposed
approach significantly improves class coverage, i.e., the number of classes
with positive recall.
