On the impossibility of non-trivial accuracy under fairness constraints
  One of the main concerns about fairness in machine learning (ML) is that, in
order to achieve it, one may have to trade off some accuracy. To overcome this
issue, Hardt et al. proposed the notion of equality of opportunity (EO), which
is compatible with maximal accuracy when the target label is deterministic with
respect to the input features.
  In the probabilistic case, however, the issue is more complicated: It has
been shown that under differential privacy constraints, there are data sources
for which EO can only be achieved at the total detriment of accuracy, in the
sense that a classifier that satisfies EO cannot be more accurate than a
trivial (i.e., constant) classifier. In our paper we strengthen this result by
removing the privacy constraint. Namely, we show that for certain data sources,
the most accurate classifier that satisfies EO is a trivial classifier.
Furthermore, we study the trade-off between accuracy and EO loss (opportunity
difference), and provide a sufficient condition on the data source under which
EO and non-trivial accuracy are compatible.
