Circa: Stochastic ReLUs for Private Deep Learning
  The simultaneous rise of machine learning as a service and concerns over user
privacy have increasingly motivated the need for private inference (PI). While
recent work demonstrates PI is possible using cryptographic primitives, the
computational overheads render it impractical. The community is largely
unprepared to address these overheads, as the source of slowdown in PI stems
from the ReLU operator whereas optimizations for plaintext inference focus on
optimizing FLOPs. In this paper we re-think the ReLU computation and propose
optimizations for PI tailored to properties of neural networks. Specifically,
we reformulate ReLU as an approximate sign test and introduce a novel
truncation method for the sign test that significantly reduces the cost per
ReLU. These optimizations result in a specific type of stochastic ReLU. The key
observation is that the stochastic fault behavior is well suited for the
fault-tolerant properties of neural network inference. Thus, we provide
significant savings without impacting accuracy. We collectively call the
optimizations Circa and demonstrate improvements of up to 4.7x storage and 3x
runtime over baseline implementations; we further show that Circa can be used
on top of recent PI optimizations to obtain 1.8x additional speedup.
