Coded-InvNet for Resilient Prediction Serving Systems
  Inspired by a new coded computation algorithm for invertible functions, we
propose Coded-InvNet a new approach to design resilient prediction serving
systems that can gracefully handle stragglers or node failures. Coded-InvNet
leverages recent findings in the deep learning literature such as invertible
neural networks, Manifold Mixup, and domain translation algorithms, identifying
interesting research directions that span across machine learning and systems.
Our experimental results show that Coded-InvNet can outperform existing
approaches, especially when the compute resource overhead is as low as 10%. For
instance, without knowing which of the ten workers is going to fail, our
algorithm can design a backup task so that it can correctly recover the missing
prediction result with an accuracy of 85.9%, significantly outperforming the
previous SOTA by 32.5%.
