Acceleration of Deep Neural Network Training with Resistive Cross-Point
  Devices
  In recent years, deep neural networks (DNN) have demonstrated significant
business impact in large scale analysis and classification tasks such as speech
recognition, visual object detection, pattern extraction, etc. Training of
large DNNs, however, is universally considered as time consuming and
computationally intensive task that demands datacenter-scale computational
resources recruited for many days. Here we propose a concept of resistive
processing unit (RPU) devices that can potentially accelerate DNN training by
orders of magnitude while using much less power. The proposed RPU device can
store and update the weight values locally thus minimizing data movement during
training and allowing to fully exploit the locality and the parallelism of the
training algorithm. We identify the RPU device and system specifications for
implementation of an accelerator chip for DNN training in a realistic
CMOS-compatible technology. For large DNNs with about 1 billion weights this
massively parallel RPU architecture can achieve acceleration factors of 30,000X
compared to state-of-the-art microprocessors while providing power efficiency
of 84,000 GigaOps/s/W. Problems that currently require days of training on a
datacenter-size cluster with thousands of machines can be addressed within
hours on a single RPU accelerator. A system consisted of a cluster of RPU
accelerators will be able to tackle Big Data problems with trillions of
parameters that is impossible to address today like, for example, natural
speech recognition and translation between all world languages, real-time
analytics on large streams of business and scientific data, integration and
analysis of multimodal sensory data flows from massive number of IoT (Internet
of Things) sensors.
