Stochastic First-order Methods for Convex and Nonconvex Functional
  Constrained Optimization
  Functional constrained optimization is becoming more and more important in
machine learning and operations research. Such problems have potential
applications in risk-averse machine learning, semisupervised learning, and
robust optimization among others. In this paper, we first present a novel
Constraint Extrapolation (ConEx) method for solving convex functional
constrained problems, which utilizes linear approximations of the constraint
functions to define the extrapolation (or acceleration) step. We show that this
method is a unified algorithm that achieves the best-known rate of convergence
for solving different functional constrained convex composite problems,
including convex or strongly convex, and smooth or nonsmooth problems with a
stochastic objective and/or stochastic constraints. Many of these rates of
convergence were in fact obtained for the first time in the literature. In
addition, ConEx is a single-loop algorithm that does not involve any penalty
subproblems. Contrary to existing primal-dual methods, it does not require the
projection of Lagrangian multipliers into a (possibly unknown) bounded set.
Second, for nonconvex functional constrained problems, we introduce a new
proximal point method that transforms the initial nonconvex problem into a
sequence of convex problems by adding quadratic terms to both the objective and
constraints. Under a certain MFCQ-type assumption, we establish the convergence
and rate of convergence of this method to KKT points when the convex
subproblems are solved exactly or inexactly. For large-scale and stochastic
problems, we present a more practical proximal point method in which the
approximate solutions of the subproblems are computed by the aforementioned
ConEx method. To the best of our knowledge, most of these convergence and
complexity results of the proximal point method for nonconvex problems also
seem to be new in the literature.
