Parameterless Stochastic Natural Gradient Method for Discrete
  Optimization and its Application to Hyper-Parameter Optimization for Neural
  Network
  Black box discrete optimization (BBDO) appears in wide range of engineering
tasks. Evolutionary or other BBDO approaches have been applied, aiming at
automating necessary tuning of system parameters, such as hyper parameter
tuning of machine learning based systems when being installed for a specific
task. However, automation is often jeopardized by the need of strategy
parameter tuning for BBDO algorithms. An expert with the domain knowledge must
undergo time-consuming strategy parameter tuning. This paper proposes a
parameterless BBDO algorithm based on information geometric optimization, a
recent framework for black box optimization using stochastic natural gradient.
Inspired by some theoretical implications, we develop an adaptation mechanism
for strategy parameters of the stochastic natural gradient method for discrete
search domains. The proposed algorithm is evaluated on commonly used test
problems. It is further extended to two examples of simultaneous optimization
of the hyper parameters and the connection weights of deep learning models,
leading to a faster optimization than the existing approaches without any
effort of parameter tuning.
