Exploiting Layerwise Convexity of Rectifier Networks with Sign
  Constrained Weights
  By introducing sign constraints on the weights, this paper proposes sign
constrained rectifier networks (SCRNs), whose training can be solved
efficiently by the well known majorization-minimization (MM) algorithms. We
prove that the proposed two-hidden-layer SCRNs, which exhibit negative weights
in the second hidden layer and negative weights in the output layer, are
capable of separating any two (or more) disjoint pattern sets. Furthermore, the
proposed two-hidden-layer SCRNs can decompose the patterns of each class into
several clusters so that each cluster is convexly separable from all the
patterns from the other classes. This provides a means to learn the pattern
structures and analyse the discriminant factors between different classes of
patterns.
