Deep Neural Networks as 0-1 Mixed Integer Linear Programs: A Feasibility
  Study
  Deep Neural Networks (DNNs) are very popular these days, and are the subject
of a very intense investigation. A DNN is made by layers of internal units (or
neurons), each of which computes an affine combination of the output of the
units in the previous layer, applies a nonlinear operator, and outputs the
corresponding value (also known as activation). A commonly-used nonlinear
operator is the so-called rectified linear unit (ReLU), whose output is just
the maximum between its input value and zero. In this (and other similar cases
like max pooling, where the max operation involves more than one input value),
one can model the DNN as a 0-1 Mixed Integer Linear Program (0-1 MILP) where
the continuous variables correspond to the output values of each unit, and a
binary variable is associated with each ReLU to model its yes/no nature. In
this paper we discuss the peculiarity of this kind of 0-1 MILP models, and
describe an effective bound-tightening technique intended to ease its solution.
We also present possible applications of the 0-1 MILP model arising in feature
visualization and in the construction of adversarial examples. Preliminary
computational results are reported, aimed at investigating (on small DNNs) the
computational performance of a state-of-the-art MILP solver when applied to a
known test case, namely, hand-written digit recognition.
