MLModelCI: An Automatic Cloud Platform for Efficient MLaaS
  MLModelCI provides multimedia researchers and developers with a one-stop
platform for efficient machine learning (ML) services. The system leverages
DevOps techniques to optimize, test, and manage models. It also containerizes
and deploys these optimized and validated models as cloud services (MLaaS). In
its essence, MLModelCI serves as a housekeeper to help users publish models.
The models are first automatically converted to optimized formats for
production purpose and then profiled under different settings (e.g., batch size
and hardware). The profiling information can be used as guidelines for
balancing the trade-off between performance and cost of MLaaS. Finally, the
system dockerizes the models for ease of deployment to cloud environments. A
key feature of MLModelCI is the implementation of a controller, which allows
elastic evaluation which only utilizes idle workers while maintaining online
service quality. Our system bridges the gap between current ML training and
serving systems and thus free developers from manual and tedious work often
associated with service deployment. We release the platform as an open-source
project on GitHub under Apache 2.0 license, with the aim that it will
facilitate and streamline more large-scale ML applications and research
projects.
