Metric-Optimized Example Weights
  Real-world machine learning applications often have complex test metrics, and
may have training and test data that are not identically distributed. Motivated
by known connections between complex test metrics and cost-weighted learning,
we propose addressing these issues by using a weighted loss function with a
standard loss, where the weights on the training examples are learned to
optimize the test metric on a validation set. These metric-optimized example
weights can be learned for any test metric, including black box and customized
ones for specific applications. We illustrate the performance of the proposed
method on diverse public benchmark datasets and real-world applications. We
also provide a generalization bound for the method.
