ReLU Networks as Surrogate Models in Mixed-Integer Linear Programs
  We consider the embedding of piecewise-linear deep neural networks (ReLU
networks) as surrogate models in mixed-integer linear programming (MILP)
problems. A MILP formulation of ReLU networks has recently been applied by many
authors to probe for various model properties subject to input bounds. The
formulation is obtained by programming each ReLU operator with a binary
variable and applying the big-M method. The efficiency of the formulation
hinges on the tightness of the bounds defined by the big-M values. When ReLU
networks are embedded in a larger optimization problem, the presence of output
bounds can be exploited in bound tightening. To this end, we devise and study
several bound tightening procedures that consider both input and output bounds.
Our numerical results show that bound tightening may reduce solution times
considerably, and that small-sized ReLU networks are suitable as surrogate
models in mixed-integer linear programs.
