Training Deep Neural Networks with Constrained Learning Parameters
  Today's deep learning models are primarily trained on CPUs and GPUs. Although
these models tend to have low error, they consume high power and utilize large
amount of memory owing to double precision floating point learning parameters.
Beyond the Moore's law, a significant portion of deep learning tasks would run
on edge computing systems, which will form an indispensable part of the entire
computation fabric. Subsequently, training deep learning models for such
systems will have to be tailored and adopted to generate models that have the
following desirable characteristics: low error, low memory, and low power. We
believe that deep neural networks (DNNs), where learning parameters are
constrained to have a set of finite discrete values, running on neuromorphic
computing systems would be instrumental for intelligent edge computing systems
having these desirable characteristics. To this extent, we propose the
Combinatorial Neural Network Training Algorithm (CoNNTrA), that leverages a
coordinate gradient descent-based approach for training deep learning models
with finite discrete learning parameters. Next, we elaborate on the theoretical
underpinnings and evaluate the computational complexity of CoNNTrA. As a proof
of concept, we use CoNNTrA to train deep learning models with ternary learning
parameters on the MNIST, Iris and ImageNet data sets and compare their
performance to the same models trained using Backpropagation. We use following
performance metrics for the comparison: (i) Training error; (ii) Validation
error; (iii) Memory usage; and (iv) Training time. Our results indicate that
CoNNTrA models use 32x less memory and have errors at par with the
Backpropagation models.
