Sequential Information Design: Markov Persuasion Process and Its
  Efficient Reinforcement Learning
  In today's economy, it becomes important for Internet platforms to consider
the sequential information design problem to align its long term interest with
incentives of the gig service providers. This paper proposes a novel model of
sequential information design, namely the Markov persuasion processes (MPPs),
where a sender, with informational advantage, seeks to persuade a stream of
myopic receivers to take actions that maximizes the sender's cumulative
utilities in a finite horizon Markovian environment with varying prior and
utility functions. Planning in MPPs thus faces the unique challenge in finding
a signaling policy that is simultaneously persuasive to the myopic receivers
and inducing the optimal long-term cumulative utilities of the sender.
Nevertheless, in the population level where the model is known, it turns out
that we can efficiently determine the optimal (resp. $\epsilon$-optimal) policy
with finite (resp. infinite) states and outcomes, through a modified
formulation of the Bellman equation.
  Our main technical contribution is to study the MPP under the online
reinforcement learning (RL) setting, where the goal is to learn the optimal
signaling policy by interacting with with the underlying MPP, without the
knowledge of the sender's utility functions, prior distributions, and the
Markov transition kernels. We design a provably efficient no-regret learning
algorithm, the Optimism-Pessimism Principle for Persuasion Process (OP4), which
features a novel combination of both optimism and pessimism principles. Our
algorithm enjoys sample efficiency by achieving a sublinear $\sqrt{T}$-regret
upper bound. Furthermore, both our algorithm and theory can be applied to MPPs
with large space of outcomes and states via function approximation, and we
showcase such a success under the linear setting.
