Bayesian Optimisation vs. Input Uncertainty Reduction
  Simulators often require calibration inputs estimated from real world data
and the quality of the estimate can significantly affect simulation output.
Particularly when performing simulation optimisation to find an optimal
solution, the uncertainty in the inputs significantly affects the quality of
the found solution. One remedy is to search for the solution that has the best
performance on average over the uncertain range of inputs yielding an optimal
compromise solution. We consider the more general setting where a user may
choose between either running simulations or instead collecting real world
data. A user may choose an input and a solution and observe the simulation
output, or instead query an external data source improving the input estimate
enabling the search for a more focused, less compromised solution. We
explicitly examine the trade-off between simulation and real data collection in
order to find the optimal solution of the simulator with the true inputs. Using
a value of information procedure, we propose a novel unified simulation
optimisation procedure called Bayesian Information Collection and Optimisation
(BICO) that, in each iteration, automatically determines which of the two
actions (running simulations or data collection) is more beneficial. Numerical
experiments demonstrate that the proposed algorithm is able to automatically
determine an appropriate balance between optimisation and data collection.
