Cogradient Descent for Dependable Learning
  Conventional gradient descent methods compute the gradients for multiple
variables through the partial derivative. Treating the coupled variables
independently while ignoring the interaction, however, leads to an insufficient
optimization for bilinear models. In this paper, we propose a dependable
learning based on Cogradient Descent (CoGD) algorithm to address the bilinear
optimization problem, providing a systematic way to coordinate the gradients of
coupling variables based on a kernelized projection function. CoGD is
introduced to solve bilinear problems when one variable is with sparsity
constraint, as often occurs in modern learning paradigms. CoGD can also be used
to decompose the association of features and weights, which further generalizes
our method to better train convolutional neural networks (CNNs) and improve the
model capacity. CoGD is applied in representative bilinear problems, including
image reconstruction, image inpainting, network pruning and CNN training.
Extensive experiments show that CoGD improves the state-of-the-arts by
significant margins. Code is available at
{https://github.com/bczhangbczhang/CoGD}.
