Deep Neural Network Approximation using Tensor Sketching
  Deep neural networks are powerful learning models that achieve
state-of-the-art performance on many computer vision, speech, and language
processing tasks. In this paper, we study a fundamental question that arises
when designing deep network architectures: Given a target network architecture
can we design a smaller network architecture that approximates the operation of
the target network? The question is, in part, motivated by the challenge of
parameter reduction (compression) in modern deep neural networks, as the ever
increasing storage and memory requirements of these networks pose a problem in
resource constrained environments.
  In this work, we focus on deep convolutional neural network architectures,
and propose a novel randomized tensor sketching technique that we utilize to
develop a unified framework for approximating the operation of both the
convolutional and fully connected layers. By applying the sketching technique
along different tensor dimensions, we design changes to the convolutional and
fully connected layers that substantially reduce the number of effective
parameters in a network. We show that the resulting smaller network can be
trained directly, and has a classification accuracy that is comparable to the
original network.
