Decentralized Multi-Task Learning Based on Extreme Learning Machines
  In multi-task learning (MTL), related tasks learn jointly to improve
generalization performance. To exploit the high learning speed of extreme
learning machines (ELMs), we apply the ELM framework to the MTL problem, where
the output weights of ELMs for all the tasks are learned collaboratively. We
first present the ELM based MTL problem in the centralized setting, which is
solved by the proposed MTL-ELM algorithm. Due to the fact that many data sets
of different tasks are geo-distributed, decentralized machine learning is
studied. We formulate the decentralized MTL problem based on ELM as majorized
multi-block optimization with coupled bi-convex objective functions. To solve
the problem, we propose the DMTL-ELM algorithm, which is a hybrid Jacobian and
Gauss-Seidel Proximal multi-block alternating direction method of multipliers
(ADMM). Further, to reduce the computation load of DMTL-ELM, DMTL-ELM with
first-order approximation (FO-DMTL-ELM) is presented. Theoretical analysis
shows that the convergence to the stationary point of DMTL-ELM and FO-DMTL-ELM
can be guaranteed conditionally. Through simulations, we demonstrate the
convergence of proposed MTL-ELM, DMTL-ELM, and FO-DMTL-ELM algorithms, and also
show that they can outperform existing MTL methods. Moreover, by adjusting the
dimension of hidden feature space, there exists a trade-off between
communication load and learning accuracy for DMTL-ELM.
