An experiment on the mechanisms of racial bias in ML-based credit
  scoring in Brazil
  We dissect an experimental credit scoring model developed with real data and
demonstrate - without access to protected attributes - how the use of location
information introduces racial bias. We analyze the tree gradient boosting model
with the aid of a game-theoretic inspired machine learning explainability
technique, counterfactual experiments and Brazilian census data. By exposing
algorithmic racial bias explaining the trained machine learning model inner
mechanisms, this experiment comprises an interesting artifact to aid the
endeavor of theoretical understanding of the emergence of racial bias in
machine learning systems. Without access to individuals' racial categories, we
show how classification parity measures using geographically defined groups
could carry information about model racial bias. The experiment testifies to
the need for methods and language that do not presuppose access to protected
attributes when auditing ML models, the importance of considering regional
specifics when addressing racial issues, and the central role of census data in
the AI research community. To the best of our knowledge, this is the first
documented case of algorithmic racial bias in ML-based credit scoring in
Brazil, the country with the second largest Black population in the world.
