Differentially Private Reinforcement Learning with Linear Function
  Approximation
  Motivated by the wide adoption of reinforcement learning (RL) in real-world
personalized services, where users' sensitive and private information needs to
be protected, we study regret minimization in finite-horizon Markov decision
processes (MDPs) under the constraints of differential privacy (DP). Compared
to existing private RL algorithms that work only on tabular finite-state,
finite-actions MDPs, we take the first step towards privacy-preserving learning
in MDPs with large state and action spaces. Specifically, we consider MDPs with
linear function approximation (in particular linear mixture MDPs) under the
notion of joint differential privacy (JDP), where the RL agent is responsible
for protecting users' sensitive data. We design two private RL algorithms that
are based on value iteration and policy optimization, respectively, and show
that they enjoy sub-linear regret performance while guaranteeing privacy
protection. Moreover, the regret bounds are independent of the number of
states, and scale at most logarithmically with the number of actions, making
the algorithms suitable for privacy protection in nowadays large-scale
personalized services. Our results are achieved via a general procedure for
learning in linear mixture MDPs under changing regularizers, which not only
generalizes previous results for non-private learning, but also serves as a
building block for general private reinforcement learning.
