Biased Stochastic Gradient Descent for Conditional Stochastic
  Optimization
  Conditional Stochastic Optimization (CSO) covers a variety of applications
ranging from meta-learning and causal inference to invariant learning. However,
constructing unbiased gradient estimates in CSO is challenging due to the
composition structure. As an alternative, we propose a biased stochastic
gradient descent (BSGD) algorithm and study the bias-variance tradeoff under
different structural assumptions. We establish the sample complexities of BSGD
for strongly convex, convex, and weakly convex objectives, under smooth and
non-smooth conditions. We also provide matching lower bounds of BSGD for convex
CSO objectives. Extensive numerical experiments are conducted to illustrate the
performance of BSGD on robust logistic regression, model-agnostic meta-learning
(MAML), and instrumental variable regression (IV).
