Max-Utility Based Arm Selection Strategy For Sequential Query
  Recommendations
  We consider the query recommendation problem in closed loop interactive
learning settings like online information gathering and exploratory analytics.
The problem can be naturally modelled using the Multi-Armed Bandits (MAB)
framework with countably many arms. The standard MAB algorithms for countably
many arms begin with selecting a random set of candidate arms and then applying
standard MAB algorithms, e.g., UCB, on this candidate set downstream. We show
that such a selection strategy often results in higher cumulative regret and to
this end, we propose a selection strategy based on the maximum utility of the
arms. We show that in tasks like online information gathering, where sequential
query recommendations are employed, the sequences of queries are correlated and
the number of potentially optimal queries can be reduced to a manageable size
by selecting queries with maximum utility with respect to the currently
executing query. Our experimental results using a recent real online literature
discovery service log file demonstrate that the proposed arm selection strategy
improves the cumulative regret substantially with respect to the
state-of-the-art baseline algorithms. % and commonly used random selection
strategy for a variety of contextual multi-armed bandit algorithms. Our data
model and source code are available at
~\url{https://anonymous.4open.science/r/0e5ad6b7-ac02-4577-9212-c9d505d3dbdb/}.
