A Survey of Optimization Methods from a Machine Learning Perspective
  Machine learning develops rapidly, which has made many theoretical
breakthroughs and is widely applied in various fields. Optimization, as an
important part of machine learning, has attracted much attention of
researchers. With the exponential growth of data amount and the increase of
model complexity, optimization methods in machine learning face more and more
challenges. A lot of work on solving optimization problems or improving
optimization methods in machine learning has been proposed successively. The
systematic retrospect and summary of the optimization methods from the
perspective of machine learning are of great significance, which can offer
guidance for both developments of optimization and machine learning research.
In this paper, we first describe the optimization problems in machine learning.
Then, we introduce the principles and progresses of commonly used optimization
methods. Next, we summarize the applications and developments of optimization
methods in some popular machine learning fields. Finally, we explore and give
some challenges and open problems for the optimization in machine learning.
