Variational Bayesian Reinforcement Learning with Regret Bounds
  In reinforcement learning the Q-values summarize the expected future rewards
that the agent will attain. However, they cannot capture the epistemic
uncertainty about those rewards. In this work we derive a new Bellman operator
with associated fixed point we call the `knowledge values'. These K-values
compress both the expected future rewards and the epistemic uncertainty into a
single value, so that high uncertainty, high reward, or both, can yield high
K-values. The key principle is to endow the agent with a risk-seeking utility
function that is carefully tuned to balance exploration and exploitation. When
the agent follows a Boltzmann policy over the K-values it yields a Bayes regret
bound of $\tilde O(L^{3/2} \sqrt{S A T})$, where $L$ is the time horizon, $S$
is the number of states, $A$ is the number of actions, and $T$ is the total
number of elapsed timesteps. We show deep connections of this approach to the
soft-max and maximum-entropy strands of research in reinforcement learning.
