Reducing Parameter Space for Neural Network Training
  For neural networks (NNs) with rectified linear unit (ReLU) or binary
activation functions, we show that their training can be accomplished in a
reduced parameter space. Specifically, the weights in each neuron can be
trained on the unit sphere, as opposed to the entire space, and the threshold
can be trained in a bounded interval, as opposed to the real line. We show that
the NNs in the reduced parameter space are mathematically equivalent to the
standard NNs with parameters in the whole space. The reduced parameter space
shall facilitate the optimization procedure for the network training, as the
search space becomes (much) smaller. We demonstrate the improved training
performance using numerical examples.
