Branching Reinforcement Learning
  In this paper, we propose a novel Branching Reinforcement Learning (Branching
RL) model, and investigate both Regret Minimization (RM) and Reward-Free
Exploration (RFE) metrics for this model. Unlike standard RL where the
trajectory of each episode is a single $H$-step path, branching RL allows an
agent to take multiple base actions in a state such that transitions branch out
to multiple successor states correspondingly, and thus it generates a
tree-structured trajectory. This model finds important applications in
hierarchical recommendation systems and online advertising. For branching RL,
we establish new Bellman equations and key lemmas, i.e., branching value
difference lemma and branching law of total variance, and also bound the total
variance by only $O(H^2)$ under an exponentially-large trajectory. For RM and
RFE metrics, we propose computationally efficient algorithms BranchVI and
BranchRFE, respectively, and derive nearly matching upper and lower bounds. Our
results are only polynomial in problem parameters despite exponentially-large
trajectories.
