ReLU activated Multi-Layer Neural Networks trained with Mixed Integer
  Linear Programs
  In this paper, it is demonstrated through a case study that multilayer
feedforward neural networks activated by ReLU functions can in principle be
trained iteratively with Mixed Integer Linear Programs (MILPs) as follows.
Weights are determined with batch learning. Multiple iterations are used per
batch of training data. In each iteration, the algorithm starts at the output
layer and propagates information back to the first hidden layer to adjust the
weights using MILPs or Linear Programs. For each layer, the goal is to minimize
the difference between its output and the corresponding target output. The
target output of the last (output) layer is equal to the ground truth. The
target output of a previous layer is defined as the adjusted input of the
following layer. For a given layer, weights are computed by solving a MILP.
Then, except for the first hidden layer, the input values are also modified
with a MILP to better match the layer outputs to their corresponding target
outputs. The method was tested and compared with Tensorflow/Keras (Adam
optimizer) using two simple networks on the MNIST dataset containing
handwritten digits. Accuracies of the same magnitude as with Tensorflow/Keras
were achieved.
