Automatic Fairness Testing of Neural Classifiers through Adversarial
  Sampling
  Although deep learning has demonstrated astonishing performance in many
applications, there are still concerns about its dependability. One desirable
property of deep learning applications with societal impact is fairness (i.e.,
non-discrimination). Unfortunately, discrimination might be intrinsically
embedded into the models due to the discrimination in the training data. As a
countermeasure, fairness testing systemically identifies discriminatory
samples, which can be used to retrain the model and improve the model's
fairness. Existing fairness testing approaches however have two major
limitations. Firstly, they only work well on traditional machine learning
models and have poor performance (e.g., effectiveness and efficiency) on deep
learning models. Secondly, they only work on simple structured (e.g., tabular)
data and are not applicable for domains such as text. In this work, we bridge
the gap by proposing a scalable and effective approach for systematically
searching for discriminatory samples while extending existing fairness testing
approaches to address a more challenging domain, i.e., text classification.
Compared with state-of-the-art methods, our approach only employs lightweight
procedures like gradient computation and clustering, which is significantly
more scalable and effective. Experimental results show that on average, our
approach explores the search space much more effectively (9.62 and 2.38 times
more than the state-of-the-art methods respectively on tabular and text
datasets) and generates much more discriminatory samples (24.95 and 2.68 times)
within a same reasonable time. Moreover, the retrained models reduce
discrimination by 57.2% and 60.2% respectively on average.
