Neural networks with linear threshold activations: structure and
  algorithms
  In this article we present new results on neural networks with linear
threshold activation functions. We precisely characterize the class of
functions that are representable by such neural networks and show that 2 hidden
layers are necessary and sufficient to represent any function representable in
the class. This is a surprising result in the light of recent exact
representability investigations for neural networks using other popular
activation functions like rectified linear units (ReLU). We also give precise
bounds on the sizes of the neural networks required to represent any function
in the class. Finally, we design an algorithm to solve the empirical risk
minimization (ERM) problem to global optimality for these neural networks with
a fixed architecture. The algorithm's running time is polynomial in the size of
the data sample, if the input dimension and the size of the network
architecture are considered fixed constants. The algorithm is unique in the
sense that it works for any architecture with any number of layers, whereas
previous polynomial time globally optimal algorithms work only for very
restricted classes of architectures.
