Probabilistic Contraction Analysis of Iterated Random Operators
  Consider a contraction operator $T$ over a complete metric space $\mathcal X$
with the fixed point $x^\star$. In many computational applications, it is
difficult to compute $T(x)$; therefore, one replaces the application
contraction operator $T$ at iteration $k$ by a random operator $\hat T^n_k$
using $n$ independent and identically distributed samples of a random variable.
Consider the Markov chain $(\hat X^n_k)_{k\in\mathbb{N}}$, which is generated
by $\hat X^n_{k+1} = \hat T^n_k(\hat X^n_k)$. In this paper, we identify some
sufficient conditions under which (i) the distribution of $\hat X^n_k$
converges to a Dirac mass over $x^\star$ as $k$ and $n$ go to infinity, and
(ii) the probability that $\hat X^n_k$ is far from $x^\star$ as $k$ goes to
infinity can be made arbitrarily small by an appropriate choice of $n$. We also
derive an upper bound on the probability that $\hat X^n_k$ is far from
$x^\star$ as $k\rightarrow \infty$. We apply the result to study the
convergence in probability of iterates generated by empirical value iteration
algorithms for discounted and average cost Markov decision problems.
