Optimization on Product Submanifolds of Convolution Kernels
  Recent advances in optimization methods used for training convolutional
neural networks (CNNs) with kernels, which are normalized according to
particular constraints, have shown remarkable success. This work introduces an
approach for training CNNs using ensembles of joint spaces of kernels
constructed using different constraints. For this purpose, we address a problem
of optimization on ensembles of products of submanifolds (PEMs) of convolution
kernels. To this end, we first propose three strategies to construct ensembles
of PEMs in CNNs. Next, we expound their geometric properties (metric and
curvature properties) in CNNs. We make use of our theoretical results by
developing a geometry-aware SGD algorithm (G-SGD) for optimization on ensembles
of PEMs to train CNNs. Moreover, we analyze convergence properties of G-SGD
considering geometric properties of PEMs. In the experimental analyses, we
employ G-SGD to train CNNs on Cifar-10, Cifar-100 and Imagenet datasets. The
results show that geometric adaptive step size computation methods of G-SGD can
improve training loss and convergence properties of CNNs. Moreover, we observe
that classification performance of baseline CNNs can be boosted using G-SGD on
ensembles of PEMs identified by multiple constraints.
