Reinforcement learning based adaptive metaheuristics
Parameter adaptation, that is the capability to automatically adjust an
algorithm's hyperparameters depending on the problem being faced, is one of the
main trends in evolutionary computation applied to numerical optimization.
While several handcrafted adaptation policies have been proposed over the years
to address this problem, only few attempts have been done so far at apply
machine learning to learn such policies. Here, we introduce a general-purpose
framework for performing parameter adaptation in continuous-domain
metaheuristics based on state-of-the-art reinforcement learning algorithms. We
demonstrate the applicability of this framework on two algorithms, namely
Covariance Matrix Adaptation Evolution Strategies (CMA-ES) and Differential
Evolution (DE), for which we learn, respectively, adaptation policies for the
step-size (for CMA-ES), and the scale factor and crossover rate (for DE). We
train these policies on a set of 46 benchmark functions at different
dimensionalities, with various inputs to the policies, in two settings: one
policy per function, and one global policy for all functions. Compared,
respectively, to the Cumulative Step-size Adaptation (CSA) policy and to two
well-known adaptive DE variants (iDE and jDE), our policies are able to produce
competitive results in the majority of cases, especially in the case of DE.