BlueFog: Make Decentralized Algorithms Practical for Optimization and
  Deep Learning
  Decentralized algorithm is a form of computation that achieves a global goal
through local dynamics that relies on low-cost communication between
directly-connected agents. On large-scale optimization tasks involving
distributed datasets, decentralized algorithms have shown strong, sometimes
superior, performance over distributed algorithms with a central node.
Recently, developing decentralized algorithms for deep learning has attracted
great attention. They are considered as low-communication-overhead alternatives
to those using a parameter server or the Ring-Allreduce protocol. However, the
lack of an easy-to-use and efficient software package has kept most
decentralized algorithms merely on paper. To fill the gap, we introduce
BlueFog, a python library for straightforward, high-performance implementations
of diverse decentralized algorithms. Based on a unified abstraction of various
communication operations, BlueFog offers intuitive interfaces to implement a
spectrum of decentralized algorithms, from those using a static, undirected
graph for synchronous operations to those using dynamic and directed graphs for
asynchronous operations. BlueFog also adopts several system-level acceleration
techniques to further optimize the performance on the deep learning tasks. On
mainstream DNN training tasks, BlueFog reaches a much higher throughput and
achieves an overall $1.2\times \sim 1.8\times$ speedup over Horovod, a
state-of-the-art distributed deep learning package based on Ring-Allreduce.
BlueFog is open source at https://github.com/Bluefog-Lib/bluefog.
