Distributed Learning in the Non-Convex World: From Batch to Streaming
  Data, and Beyond
  Distributed learning has become a critical enabler of the massively connected
world envisioned by many. This article discusses four key elements of scalable
distributed processing and real-time intelligence --- problems, data,
communication and computation. Our aim is to provide a fresh and unique
perspective about how these elements should work together in an effective and
coherent manner. In particular, we {provide a selective review} about the
recent techniques developed for optimizing non-convex models (i.e., problem
classes), processing batch and streaming data (i.e., data types), over the
networks in a distributed manner (i.e., communication and computation
paradigm). We describe the intuitions and connections behind a core set of
popular distributed algorithms, emphasizing how to trade off between
computation and communication costs. Practical issues and future research
directions will also be discussed.
