Fairness in the Eyes of the Data: Certifying Machine-Learning Models
  We present a framework that allows to certify the fairness degree of a model
based on an interactive and privacy-preserving test. The framework verifies any
trained model, regardless of its training process and architecture. Thus, it
allows us to evaluate any deep learning model on multiple fairness definitions
empirically. We tackle two scenarios, where either the test data is privately
available only to the tester or is publicly known in advance, even to the model
creator. We investigate the soundness of the proposed approach using
theoretical analysis and present statistical guarantees for the interactive
test. Finally, we provide a cryptographic technique to automate fairness
testing and certified inference with only black-box access to the model at hand
while hiding the participants' sensitive data.
