Measure Twice, Cut Once: Quantifying Bias and Fairness in Deep Neural
  Networks
  Algorithmic bias is of increasing concern, both to the research community,
and society at large. Bias in AI is more abstract and unintuitive than
traditional forms of discrimination and can be more difficult to detect and
mitigate. A clear gap exists in the current literature on evaluating the
relative bias in the performance of multi-class classifiers. In this work, we
propose two simple yet effective metrics, Combined Error Variance (CEV) and
Symmetric Distance Error (SDE), to quantitatively evaluate the class-wise bias
of two models in comparison to one another. By evaluating the performance of
these new metrics and by demonstrating their practical application, we show
that they can be used to measure fairness as well as bias. These demonstrations
show that our metrics can address specific needs for measuring bias in
multi-class classification.
