Towards Quantized Model Parallelism for Graph-Augmented MLPs Based on
  Gradient-Free ADMM framework
  The Graph Augmented Multi-layer Perceptron (GA-MLP) model is an attractive
alternative to Graph Neural Networks (GNNs). This is because it is resistant to
the over-smoothing problem, and deeper GA-MLP models yield better performance.
GA-MLP models are traditionally optimized by the Stochastic Gradient Descent
(SGD). However, SGD suffers from the layer dependency problem, which prevents
the gradients of different layers of GA-MLP models from being calculated in
parallel. In this paper, we propose a parallel deep learning Alternating
Direction Method of Multipliers (pdADMM) framework to achieve model
parallelism: parameters in each layer of GA-MLP models can be updated in
parallel. The extended pdADMM-Q algorithm reduces communication cost by
utilizing the quantization technique. Theoretical convergence to a critical
point of the pdADMM algorithm and the pdADMM-Q algorithm is provided with a
sublinear convergence rate $o(1/k)$. Extensive experiments in six benchmark
datasets demonstrate that the pdADMM can lead to high speedup, and outperforms
all the existing state-of-the-art comparison methods.
