Self-organizing Democratized Learning: Towards Large-scale Distributed
  Learning Systems
  Emerging cross-device artificial intelligence (AI) applications require a
transition from conventional centralized learning systems towards large-scale
distributed AI systems that can collaboratively perform complex learning tasks.
In this regard, democratized learning (Dem-AI) lays out a holistic philosophy
with underlying principles for building large-scale distributed and
democratized machine learning systems. The outlined principles are meant to
study a generalization in distributed learning systems that goes beyond
existing mechanisms such as federated learning. Moreover, such learning systems
rely on hierarchical self-organization of well-connected distributed learning
agents who have limited and highly personalized data and can evolve and
regulate themselves based on the underlying duality of specialized and
generalized processes. Inspired by Dem-AI philosophy, a novel distributed
learning approach is proposed in this paper. The approach consists of a
self-organizing hierarchical structuring mechanism based on agglomerative
clustering, hierarchical generalization, and corresponding learning mechanism.
Subsequently, hierarchical generalized learning problems in recursive forms are
formulated and shown to be approximately solved using the solutions of
distributed personalized learning problems and hierarchical update mechanisms.
To that end, a distributed learning algorithm, namely DemLearn is proposed.
Extensive experiments on benchmark MNIST, Fashion-MNIST, FE-MNIST, and CIFAR-10
datasets show that the proposed algorithms demonstrate better results in the
generalization performance of learning models in agents compared to the
conventional FL algorithms. The detailed analysis provides useful observations
to further handle both the generalization and specialization performance of the
learning models in Dem-AI systems.
