Towards Understanding Gender Bias in Relation Extraction
  Recent developments in Neural Relation Extraction (NRE) have made significant
strides towards Automated Knowledge Base Construction (AKBC). While much
attention has been dedicated towards improvements in accuracy, there have been
no attempts in the literature to our knowledge to evaluate social biases in NRE
systems. We create WikiGenderBias, a distantly supervised dataset with a human
annotated test set. WikiGenderBias has sentences specifically curated to
analyze gender bias in relation extraction systems. We use WikiGenderBias to
evaluate systems for bias and find that NRE systems exhibit gender biased
predictions and lay groundwork for future evaluation of bias in NRE. We also
analyze how name anonymization, hard debiasing for word embeddings, and
counterfactual data augmentation affect gender bias in predictions and
performance.
