Fair Machine Learning under Limited Demographically Labeled Data
  Research has shown that, machine learning models might inherit and propagate
undesired social biases encoded in the data. To address this problem, fair
training algorithms are developed. However, most algorithms assume we know
demographic/sensitive data features such as gender and race. This assumption
falls short in scenarios where collecting demographic information is not
feasible due to privacy concerns, and data protection policies. A recent line
of work develops fair training methods that can function without any
demographic feature on the data, that are collectively referred as Rawlsian
methods. Yet, we show in experiments that, Rawlsian methods tend to exhibit
relatively high bias. Given this, we look at the middle ground between the
previous approaches, and consider a setting where we know the demographic
attributes for only a small subset of our data. In such a setting, we design
fair training algorithms which exhibit both good utility, and low bias. In
particular, we show that our techniques can train models to significantly
outperform Rawlsian approaches even when 0.1% of demographic attributes are
available in the training data. Furthermore, our main algorithm can accommodate
multiple training objectives easily. We expand our main algorithm to achieve
robustness to label noise in addition to fairness in the limited demographics
setting to highlight that property as well.
