A discrete version of CMA-ES
  Modern machine learning uses more and more advanced optimization techniques
to find optimal hyper parameters. Whenever the objective function is
non-convex, non continuous and with potentially multiple local minima, standard
gradient descent optimization methods fail. A last resource and very different
method is to assume that the optimum(s), not necessarily unique, is/are
distributed according to a distribution and iteratively to adapt the
distribution according to tested points. These strategies originated in the
early 1960s, named Evolution Strategy (ES) have culminated with the CMA-ES
(Covariance Matrix Adaptation) ES. It relies on a multi variate normal
distribution and is supposed to be state of the art for general optimization
program. However, it is far from being optimal for discrete variables. In this
paper, we extend the method to multivariate binomial correlated distributions.
For such a distribution, we show that it shares similar features to the multi
variate normal: independence and correlation is equivalent and correlation is
efficiently modeled by interaction between different variables. We discuss this
distribution in the framework of the exponential family. We prove that the
model can estimate not only pairwise interactions among the two variables but
also is capable of modeling higher order interactions. This allows creating a
version of CMA ES that can accommodate efficiently discrete variables. We
provide the corresponding algorithm and conclude.
