State Action Separable Reinforcement Learning
  Reinforcement Learning (RL) based methods have seen their paramount successes
in solving serial decision-making and control problems in recent years. For
conventional RL formulations, Markov Decision Process (MDP) and
state-action-value function are the basis for the problem modeling and policy
evaluation. However, several challenging issues still remain. Among most cited
issues, the enormity of state/action space is an important factor that causes
inefficiency in accurately approximating the state-action-value function. We
observe that although actions directly define the agents' behaviors, for many
problems the next state after a state transition matters more than the action
taken, in determining the return of such a state transition. In this regard, we
propose a new learning paradigm, State Action Separable Reinforcement Learning
(sasRL), wherein the action space is decoupled from the value function learning
process for higher efficiency. Then, a light-weight transition model is learned
to assist the agent to determine the action that triggers the associated state
transition. In addition, our convergence analysis reveals that under certain
conditions, the convergence time of sasRL is $O(T^{1/k})$, where $T$ is the
convergence time for updating the value function in the MDP-based formulation
and $k$ is a weighting factor. Experiments on several gaming scenarios show
that sasRL outperforms state-of-the-art MDP-based RL algorithms by up to
$75\%$.
