SoK: Privacy-preserving Deep Learning with Homomorphic Encryption
  Outsourced computation for neural networks allows users access to state of
the art models without needing to invest in specialized hardware and know-how.
The problem is that the users lose control over potentially privacy sensitive
data. With homomorphic encryption (HE) computation can be performed on
encrypted data without revealing its content. In this systematization of
knowledge, we take an in-depth look at approaches that combine neural networks
with HE for privacy preservation. We categorize the changes to neural network
models and architectures to make them computable over HE and how these changes
impact performance. We find numerous challenges to HE based privacy-preserving
deep learning such as computational overhead, usability, and limitations posed
by the encryption schemes.
