Compressed Learning of Deep Neural Networks for OpenCL-Capable Embedded
  Systems
  Deep neural networks (DNNs) have been quite successful in solving many
complex learning problems. However, DNNs tend to have a large number of
learning parameters, leading to a large memory and computation requirement. In
this paper, we propose a model compression framework for efficient training and
inference of deep neural networks on embedded systems. Our framework provides
data structures and kernels for OpenCL-based parallel forward and backward
computation in a compressed form. In particular, our method learns sparse
representations of parameters using $\ell_1$-based sparse coding while
training, storing them in compressed sparse matrices. Unlike the previous
works, our method does not require a pre-trained model as an input and
therefore can be more versatile for different application environments. Even
though the use of $\ell_1$-based sparse coding for model compression is not
new, we show that it can be far more effective than previously reported when we
use proximal point algorithms and the technique of debiasing. Our experiments
show that our method can produce minimal learning models suitable for small
embedded devices.
