Pulling Up by the Causal Bootstraps: Causal Data Augmentation for
  Pre-training Debiasing
  Machine learning models achieve state-of-the-art performance on many
supervised learning tasks. However, prior evidence suggests that these models
may learn to rely on shortcut biases or spurious correlations (intuitively,
correlations that do not hold in the test as they hold in train) for good
predictive performance. Such models cannot be trusted in deployment
environments to provide accurate predictions. While viewing the problem from a
causal lens is known to be useful, the seamless integration of causation
techniques into machine learning pipelines remains cumbersome and expensive. In
this work, we study and extend a causal pre-training debiasing technique called
causal bootstrapping (CB) under five practical confounded-data
generation-acquisition scenarios (with known and unknown confounding). Under
these settings, we systematically investigate the effect of confounding bias on
deep learning model performance, demonstrating their propensity to rely on
shortcut biases when these biases are not properly accounted for. We
demonstrate that such a causal pre-training technique can significantly
outperform existing base practices to mitigate confounding bias on real-world
domain generalization benchmarking tasks. This systematic investigation
underlines the importance of accounting for the underlying data-generating
mechanisms and fortifying data-preprocessing pipelines with a causal framework
to develop methods robust to confounding biases.
