Under the Radar -- Auditing Fairness in ML for Humanitarian Mapping
  Humanitarian mapping from space with machine learning helps policy-makers to
timely and accurately identify people in need. However, recent concerns around
fairness and transparency of algorithmic decision-making are a significant
obstacle for applying these methods in practice. In this paper, we study if
humanitarian mapping approaches from space are prone to bias in their
predictions. We map village-level poverty and electricity rates in India based
on nighttime lights (NTLs) with linear regression and random forest and analyze
if the predictions systematically show prejudice against scheduled caste or
tribe communities. To achieve this, we design a causal approach to measure
counterfactual fairness based on propensity score matching. This allows to
compare villages within a community of interest to synthetic counterfactuals.
Our findings indicate that poverty is systematically overestimated and
electricity systematically underestimated for scheduled tribes in comparison to
a synthetic counterfactual group of villages. The effects have the opposite
direction for scheduled castes where poverty is underestimated and
electrification overestimated. These results are a warning sign for a variety
of applications in humanitarian mapping where fairness issues would compromise
policy goals.
