BigDL 2.0: Seamless Scaling of AI Pipelines from Laptops to Distributed
  Cluster
  Most AI projects start with a Python notebook running on a single laptop;
however, one usually needs to go through a mountain of pains to scale it to
handle larger dataset (for both experimentation and production deployment).
These usually entail many manual and error-prone steps for the data scientists
to fully take advantage of the available hardware resources (e.g., SIMD
instructions, multi-processing, quantization, memory allocation optimization,
data partitioning, distributed computing, etc.). To address this challenge, we
have open sourced BigDL 2.0 at https://github.com/intel-analytics/BigDL/ under
Apache 2.0 license (combining the original BigDL and Analytics Zoo projects);
using BigDL 2.0, users can simply build conventional Python notebooks on their
laptops (with possible AutoML support), which can then be transparently
accelerated on a single node (with up-to 9.6x speedup in our experiments), and
seamlessly scaled out to a large cluster (across several hundreds servers in
real-world use cases). BigDL 2.0 has already been adopted by many real-world
users (such as Mastercard, Burger King, Inspur, etc.) in production.
