Input Invex Neural Network
  In this paper, we present a novel method to constrain invexity on Neural
Networks (NN). Invex functions ensure every stationary point is global minima.
Hence, gradient descent commenced from any point will lead to the global
minima. Another advantage of invexity on NN is to divide data space locally
into two connected sets with a highly non-linear decision boundary by simply
thresholding the output. To this end, we formulate a universal invex function
approximator and employ it to enforce invexity in NN. We call it Input Invex
Neural Networks (II-NN). We first fit data with a known invex function,
followed by modification with a NN, compare the direction of the gradient and
penalize the direction of gradient on NN if it contradicts with the direction
of reference invex function. In order to penalize the direction of the gradient
we perform Gradient Clipped Gradient Penalty (GC-GP). We applied our method to
the existing NNs for both image classification and regression tasks. From the
extensive empirical and qualitative experiments, we observe that our method
gives the performance similar to ordinary NN yet having invexity. Our method
outperforms linear NN and Input Convex Neural Network (ICNN) with a large
margin. We publish our code and implementation details at github.
