An LP-based hyperparameter optimization model for language modeling
  In order to find hyperparameters for a machine learning model, algorithms
such as grid search or random search are used over the space of possible values
of the models hyperparameters. These search algorithms opt the solution that
minimizes a specific cost function. In language models, perplexity is one of
the most popular cost functions. In this study, we propose a fractional
nonlinear programming model that finds the optimal perplexity value. The
special structure of the model allows us to approximate it by a linear
programming model that can be solved using the well-known simplex algorithm. To
the best of our knowledge, this is the first attempt to use optimization
techniques to find perplexity values in the language modeling literature. We
apply our model to find hyperparameters of a language model and compare it to
the grid search algorithm. Furthermore, we illustrating that it results in
lower perplexity values. We perform this experiment on a real-world dataset
from SwiftKey to validate our proposed approach.
