Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow
  In the past decade, model-free reinforcement learning (RL) has provided
solutions to challenging domains such as robotics. Model-based RL shows the
prospect of being more sample-efficient than model-free methods in terms of
agent-environment interactions, because the model enables to extrapolate to
unseen situations. In the more recent past, model-based methods have shown
superior results compared to model-free methods in some challenging domains
with non-linear state transitions. At the same time, it has become apparent
that RL is not market-ready yet and that many real-world applications are going
to require model-based approaches, because model-free methods are too
sample-inefficient and show poor performance in early stages of training. The
latter is particularly important in industry, e.g. in production systems that
directly impact a company's revenue. This demonstrates the necessity for a
toolbox to push the boundaries for model-based RL. While there is a plethora of
toolboxes for model-free RL, model-based RL has received little attention in
terms of toolbox development. Bellman aims to fill this gap and introduces the
first thoroughly designed and tested model-based RL toolbox using
state-of-the-art software engineering practices. Our modular approach enables
to combine a wide range of environment models with generic model-based agent
classes that recover state-of-the-art algorithms. We also provide an experiment
harness to compare both model-free and model-based agents in a systematic
fashion w.r.t. user-defined evaluation metrics (e.g. cumulative reward). This
paves the way for new research directions, e.g. investigating uncertainty-aware
environment models that are not necessarily neural-network-based, or developing
algorithms to solve industrially-motivated benchmarks that share
characteristics with real-world problems.
