Mixing ADAM and SGD: a Combined Optimization Method
  Optimization methods (optimizers) get special attention for the efficient
training of neural networks in the field of deep learning. In literature there
are many papers that compare neural models trained with the use of different
optimizers. Each paper demonstrates that for a particular problem an optimizer
is better than the others but as the problem changes this type of result is no
longer valid and we have to start from scratch. In our paper we propose to use
the combination of two very different optimizers but when used simultaneously
they can overcome the performances of the single optimizers in very different
problems. We propose a new optimizer called MAS (Mixing ADAM and SGD) that
integrates SGD and ADAM simultaneously by weighing the contributions of both
through the assignment of constant weights. Rather than trying to improve SGD
or ADAM we exploit both at the same time by taking the best of both. We have
conducted several experiments on images and text document classification, using
various CNNs, and we demonstrated by experiments that the proposed MAS
optimizer produces better performance than the single SGD or ADAM optimizers.
The source code and all the results of the experiments are available online at
the following link https://gitlab.com/nicolalandro/multi\_optimizer
