Energy-Constrained Compression for Deep Neural Networks via Weighted
  Sparse Projection and Layer Input Masking
  Deep Neural Networks (DNNs) are increasingly deployed in highly
energy-constrained environments such as autonomous drones and wearable devices
while at the same time must operate in real-time. Therefore, reducing the
energy consumption has become a major design consideration in DNN training.
This paper proposes the first end-to-end DNN training framework that provides
quantitative energy consumption guarantees via weighted sparse projection and
input masking. The key idea is to formulate the DNN training as an optimization
problem in which the energy budget imposes a previously unconsidered
optimization constraint. We integrate the quantitative DNN energy estimation
into the DNN training process to assist the constrained optimization. We prove
that an approximate algorithm can be used to efficiently solve the optimization
problem. Compared to the best prior energy-saving methods, our framework trains
DNNs that provide higher accuracies under same or lower energy budgets. Code is
publicly available.
