TSEC: a framework for online experimentation under experimental
  constraints
  Thompson sampling is a popular algorithm for solving multi-armed bandit
problems, and has been applied in a wide range of applications, from website
design to portfolio optimization. In such applications, however, the number of
choices (or arms) $N$ can be large, and the data needed to make adaptive
decisions require expensive experimentation. One is then faced with the
constraint of experimenting on only a small subset of $K \ll N$ arms within
each time period, which poses a problem for traditional Thompson sampling. We
propose a new Thompson Sampling under Experimental Constraints (TSEC) method,
which addresses this so-called "arm budget constraint". TSEC makes use of a
Bayesian interaction model with effect hierarchy priors, to model correlations
between rewards on different arms. This fitted model is then integrated within
Thompson sampling, to jointly identify a good subset of arms for
experimentation and to allocate resources over these arms. We demonstrate the
effectiveness of TSEC in two problems with arm budget constraints. The first is
a simulated website optimization study, where TSEC shows noticeable
improvements over industry benchmarks. The second is a portfolio optimization
application on industry-based exchange-traded funds, where TSEC provides more
consistent and greater wealth accumulation over standard investment strategies.
