Approximate Bayesian Optimisation for Neural Networks
  A body of work has been done to automate machine learning algorithm to
highlight the importance of model choice. Automating the process of choosing
the best forecasting model and its corresponding parameters can result to
improve a wide range of real-world applications. Bayesian optimisation (BO)
uses a blackbox optimisation methods to propose solutions according to an
exploration-exploitation trade-off criterion through acquisition functions. BO
framework imposes two key ingredients: a probabilistic surrogate model that
consist of prior belief of the unknown objective function(data-dependant) and
an objective function that describes how optimal is the model-fit. Choosing the
best model and its associated hyperparameters can be very expensive, and is
typically fit using Gaussian processes (GPs) and at some extends applying
approximate inference due its intractability. However, since GPs scale
cubically with the number of observations, it has been challenging to handle
objectives whose optimization requires many evaluations. In addition, most
real-dataset are non-stationary which make idealistic assumptions on surrogate
models. The necessity to solve the analytical tractability and the
computational feasibility in a stochastic fashion enables to ensure the
efficiency and the applicability of Bayesian optimisation. In this paper we
explore the use of neural networks as an alternative to GPs to model
distributions over functions, we provide a link between density-ratio
estimation and class probability estimation based on approximate inference,
this reformulation provides algorithm efficiency and tractability.
