Low-Rank Training of Deep Neural Networks for Emerging Memory Technology
  The recent success of neural networks for solving difficult decision tasks
has incentivized incorporating smart decision making "at the edge." However,
this work has traditionally focused on neural network inference, rather than
training, due to memory and compute limitations, especially in emerging
non-volatile memory systems, where writes are energetically costly and reduce
lifespan. Yet, the ability to train at the edge is becoming increasingly
important as it enables real-time adaptability to device drift and
environmental variation, user customization, and federated learning across
devices. In this work, we address two key challenges for training on edge
devices with non-volatile memory: low write density and low auxiliary memory.
We present a low-rank training scheme that addresses these challenges while
maintaining computational efficiency. We then demonstrate the technique on a
representative convolutional neural network across several adaptation problems,
where it out-performs standard SGD both in accuracy and in number of weight
writes.
