NullSpaceNet: Nullspace Convoluional Neural Network with Differentiable
  Loss Function
  We propose NullSpaceNet, a novel network that maps from the pixel level input
to a joint-nullspace (as opposed to the traditional feature space), where the
newly learned joint-nullspace features have clearer interpretation and are more
separable. NullSpaceNet ensures that all inputs from the same class are
collapsed into one point in this new joint-nullspace, and the different classes
are collapsed into different points with high separation margins. Moreover, a
novel differentiable loss function is proposed that has a closed-form solution
with no free-parameters. NullSpaceNet exhibits superior performance when tested
against VGG16 with fully-connected layer over 4 different datasets, with
accuracy gain of up to 4.55%, a reduction in learnable parameters from 135M to
19M, and reduction in inference time of 99% in favor of NullSpaceNet. This
means that NullSpaceNet needs less than 1% of the time it takes a traditional
CNN to classify a batch of images with better accuracy.
