Alice's Adventures in the Markovian World
  This paper proposes an algorithm Alice having no access to the physics law of
the environment, which is actually linear with stochastic noise, and learns to
make decisions directly online without a training phase or a stable policy as
initial input. Neither estimating the system parameters nor the value functions
online, the proposed algorithm generalizes one of the most fundamental online
learning algorithms Follow-the-Leader into a linear Gauss-Markov process
setting, with a regularization term similar to the momentum method in the
gradient descent algorithm, and a feasible online constraint inspired by
Lyapunov's Second Theorem. The proposed algorithm is considered as a mirror
optimization to the model predictive control. Only knowing the state-action
alignment relationship, with the ability to observe every state exactly, a
no-regret proof of the algorithm without state noise is given. The analysis of
the general linear system with stochastic noise is shown with a sufficient
condition for the no-regret proof. The simulations compare the performance of
Alice with another recent work and verify the great flexibility of Alice.
