ENSEI: Efficient Secure Inference via Frequency-Domain Homomorphic
  Convolution for Privacy-Preserving Visual Recognition
  In this work, we propose ENSEI, a secure inference (SI) framework based on
the frequency-domain secure convolution (FDSC) protocol for the efficient
execution of privacy-preserving visual recognition. Our observation is that,
under the combination of homomorphic encryption and secret sharing, homomorphic
convolution can be obliviously carried out in the frequency domain,
significantly simplifying the related computations. We provide protocol designs
and parameter derivations for number-theoretic transform (NTT) based FDSC. In
the experiment, we thoroughly study the accuracy-efficiency trade-offs between
time- and frequency-domain homomorphic convolution. With ENSEI, compared to the
best known works, we achieve 5--11x online time reduction, up to 33x setup time
reduction, and up to 10x reduction in the overall inference time. A further 33%
of bandwidth reductions can be obtained on binary neural networks with only 1%
of accuracy degradation on the CIFAR-10 dataset.
