Optimal Bidding Strategy without Exploration in Real-time Bidding
  Maximizing utility with a budget constraint is the primary goal for
advertisers in real-time bidding (RTB) systems. The policy maximizing the
utility is referred to as the optimal bidding strategy. Earlier works on
optimal bidding strategy apply model-based batch reinforcement learning methods
which can not generalize to unknown budget and time constraint. Further, the
advertiser observes a censored market price which makes direct evaluation
infeasible on batch test datasets. Previous works ignore the losing auctions to
alleviate the difficulty with censored states; thus significantly modifying the
test distribution. We address the challenge of lacking a clear evaluation
procedure as well as the error propagated through batch reinforcement learning
methods in RTB systems. We exploit two conditional independence structures in
the sequential bidding process that allow us to propose a novel practical
framework using the maximum entropy principle to imitate the behavior of the
true distribution observed in real-time traffic. Moreover, the framework allows
us to train a model that can generalize to the unseen budget conditions than
limit only to those observed in history. We compare our methods on two
real-world RTB datasets with several baselines and demonstrate significantly
improved performance under various budget settings.
