Quadratic Metric Elicitation for Fairness and Beyond
  Metric elicitation is a recent framework for eliciting performance metrics
that best reflect implicit user preferences based on the application and
context. However, available elicitation strategies have been limited to linear
(or quasi-linear) functions of predictive rates, which can be practically
restrictive for many domains including fairness. This paper develops a strategy
for eliciting more flexible multiclass metrics defined by quadratic functions
of rates, designed to reflect human preferences better. We show its application
in eliciting quadratic violation-based group-fair metrics. Our strategy
requires only relative preference feedback, and that too of near-optimal
amount, and is robust to feedback noise. We further extend this strategy to
eliciting polynomial metrics -- thus broadening the use cases for metric
elicitation.
