Sphynx: ReLU-Efficient Network Design for Private Inference
  The emergence of deep learning has been accompanied by privacy concerns
surrounding users' data and service providers' models. We focus on private
inference (PI), where the goal is to perform inference on a user's data sample
using a service provider's model. Existing PI methods for deep networks enable
cryptographically secure inference with little drop in functionality; however,
they incur severe latency costs, primarily caused by non-linear network
operations (such as ReLUs). This paper presents Sphynx, a ReLU-efficient
network design method based on micro-search strategies for convolutional cell
design. Sphynx achieves Pareto dominance over all existing private inference
methods on CIFAR-100. We also design large-scale networks that support
cryptographically private inference on Tiny-ImageNet and ImageNet.
