Exploration with Limited Memory: Streaming Algorithms for Coin Tossing,
  Noisy Comparisons, and Multi-Armed Bandits
  Consider the following abstract coin tossing problem: Given a set of $n$
coins with unknown biases, find the most biased coin using a minimal number of
coin tosses. This is a common abstraction of various exploration problems in
theoretical computer science and machine learning and has been studied
extensively over the years. In particular, algorithms with optimal sample
complexity (number of coin tosses) have been known for this problem for quite
some time.
  Motivated by applications to processing massive datasets, we study the space
complexity of solving this problem with optimal number of coin tosses in the
streaming model. In this model, the coins are arriving one by one and the
algorithm is only allowed to store a limited number of coins at any point --
any coin not present in the memory is lost and can no longer be tossed or
compared to arriving coins. Prior algorithms for the coin tossing problem with
optimal sample complexity are based on iterative elimination of coins which
inherently require storing all the coins, leading to memory-inefficient
streaming algorithms.
  We remedy this state-of-affairs by presenting a series of improved streaming
algorithms for this problem: we start with a simple algorithm which require
storing only $O(\log{n})$ coins and then iteratively refine it further and
further, leading to algorithms with $O(\log\log{(n)})$ memory, $O(\log^*{(n)})$
memory, and finally a one that only stores a single extra coin in memory -- the
same exact space needed to just store the best coin throughout the stream.
  Furthermore, we extend our algorithms to the problem of finding the $k$ most
biased coins as well as other exploration problems such as finding top-$k$
elements using noisy comparisons or finding an $\epsilon$-best arm in
stochastic multi-armed bandits, and obtain efficient streaming algorithms for
these problems.
