Assessing the Impact: Does an Improvement to a Revenue Management System
  Lead to an Improved Revenue?
  Airlines and other industries have been making use of sophisticated Revenue
Management Systems to maximize revenue for decades. While improving the
different components of these systems has been the focus of numerous studies,
estimating the impact of such improvements on the revenue has been overlooked
in the literature despite its practical importance. Indeed, quantifying the
benefit of a change in a system serves as support for investment decisions.
This is a challenging problem as it corresponds to the difference between the
generated value and the value that would have been generated keeping the system
as before. The latter is not observable. Moreover, the expected impact can be
small in relative value. In this paper, we cast the problem as counterfactual
prediction of unobserved revenue. The impact on revenue is then the difference
between the observed and the estimated revenue. The originality of this work
lies in the innovative application of econometric methods proposed for
macroeconomic applications to a new problem setting. Broadly applicable, the
approach benefits from only requiring revenue data observed for
origin-destination pairs in the network of the airline at each day, before and
after a change in the system is applied. We report results using real
large-scale data from Air Canada. We compare a deep neural network
counterfactual predictions model with econometric models. They achieve
respectively 1% and 1.1% of error on the counterfactual revenue predictions,
and allow to accurately estimate small impacts (in the order of 2%).
