{'fraction_non_alphanumeric': 0.03607042692427548, 'fraction_numerical': 0.009192894982860704, 'mean_word_length': 5.130372492836676, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow

In the past decade, model-free reinforcement learning (RL) has provided solutions to challenging domains such as robotics. Model-based RL shows the prospect of being more sample-efficient than model-free methods in terms of agent-environment interactions, because the model enables to extrapolate to unseen situations. In the more recent past, model-based methods have shown superior results compared to model-free methods in some challenging domains with non-linear state transitions. At the same time, it has become apparent that RL is not market-ready yet and that many real-world applications are going to require model-based approaches, because model-free methods are too sample-inefficient and show poor performance in early stages of training. The latter is particularly important in industry, e.g. in production systems that directly impact a company's revenue. This demonstrates the necessity for a toolbox to push the boundaries for model-based RL. While there is a plethora of toolboxes for model-free RL, model-based RL has received little attention in terms of toolbox development. Bellman aims to fill this gap and introduces the first thoroughly designed and tested model-based RL toolbox using state-of-the-art software engineering practices. Our modular approach enables to combine a wide range of environment models with generic model-based agent classes that recover state-of-the-art algorithms. We also provide an experiment harness to compare both model-free and model-based agents in a systematic fashion w.r.t. user-defined evaluation metrics (e.g. cumulative reward). This paves the way for new research directions, e.g. investigating uncertainty-aware environment models that are not necessarily neural-network-based, or developing algorithms to solve industrially-motivated benchmarks that share characteristics with real-world problems.

In the past decade, model-free reinforcement learning (RL) has provided solutions to challenging domains such as robotics. Model-based RL (where agents learn a model of the environment in order to explicitly plan ahead) shows the prospect of being more sampleefficient than model-free methods in terms of agent-environment interactions, because the model enables to extrapolate to unseen situations. In the more recent past, model-based methods have shown superior results compared to model-free methods in some challenging domains with non-linear state transitions. At the same time, it has become apparent that RL is not market-ready yet and that many real-world applications are going to require model-based approaches, because model-free methods are too sample-inefficient and show poor performance in early stages of training. The latter is particularly important in industry, e.g. in production systems that directly impact a company's revenue. This demonstrates the necessity for a toolbox to push the boundaries for model-based RL. While there is a plethora of toolboxes for model-free RL, model-based RL has received little attention in terms of toolbox development. Bellman aims to fill this gap and introduces the first thoroughly designed and tested model-based RL toolbox using state-of-the-art software engineering practices. Our modular approach enables to combine a wide range of environment models with generic model-based agent classes that recover state-of-the-art algorithms. We also provide an experiment harness to compare both model-free and model-based agents in a systematic fashion w.r.t. user-defined evaluation metrics (e.g. cumulative reward). This paves the way for new research directions, e.g. investigating uncertainty-aware environment models that are not necessarily neural-network-based, or developing algorithms to solve industrially-motivated benchmarks that share characteristics with real-world problems.

Existing Toolboxes for Reinforcement Learning
In recent years, plenty of toolboxes with state-of-the-art model-free RL algorithms have been developed. We enumerate some of them in the following but do not claim this to be an exhaustive list. Amongst the earliest were "RL-Glue" (Tanner and White, 2009), "rllab" (Duan et al., 2016) and "keras-rl" (Plappert, 2016), followed by "Reinforcement Learning Coach" (Caspi et al., 2017), "OpenAI Baselines" (Dhariwal et al., 2017) and "Tensorforce" (Kuhnle et al., 2017), that were later succeeded by "Stable Baselines" (Hill et al., 2018) and "RLlib" (Liang et al., 2018). At the time of writing, "TF-Agents" (Guadarrama et al., 2018) in TensorFlow (Abadi et al., 2016) and"Horizon" (Gauci et al., 2019) (2020) provide a software framework called "Baconian" (with some algorithms being completed and some still being developed). While the former focuses more on the research aspect and the second more on the software aspect, what both have in common is that algorithms are implemented in a stand-alone fashion in separate classes without modular & hierarchical object-oriented abstractions and without the flexibility to design new algorithms.

Motivation for a Model-Based Reinforcement Learning Toolbox
Recently, model-based RL methods have been developed that outperform model-free methods in challenging high-dimensional domains (Wang et al., 2019). At the same time, the community became aware of issues that arise when applying RL in practice, and there is an implicit understanding that many real-world problems are going to require model-based approaches (Dulac-Arnold et al., 2019) for sample-efficiency reasons and because simulators of industrial systems are not available. A second observation is that the literature presents algorithms in a standalone fashion because they are usually developed by different groups, and frameworks are designed in such a way that individual algorithms are implemented in separate classes without inheriting from well-designed object-oriented superclasses that exploit commonalities between algorithms (Wang et al., 2019;Dong et al., 2020). It is often difficult to tell which precise combination of algorithmic components yields superior results: is it the environment model, the decision-making component, or the choice of optimization objectives? The fact that different papers report results under different environments, reward functions or experiment protocols exacerbates the situation (Henderson et al., 2018).
The aim of Bellman was to address those problems via identifying commonalities between contemporary model-based RL algorithms and designing a modular software framework where algorithms arise naturally in subclasses and from the composition of well-chosen object-oriented abstractions. This enables the comparison of different algorithms, but importantly, in a controlled setup where the model-based component across different algorithms is the same. Alternatively, different models can be tested simply by changing the environment model object (e.g. by replacing a neural net with a Gaussian process). Furthermore, Bellman provides an experiment harness to train and evaluate model-based and -free algorithms systematically and user-conveniently under the same experiment protocol.

Key Features of Bellman and Implementation Details
Bellman extends the TensorFlow-based library TF-Agents (Guadarrama et al., 2018) that contains a collection of model-free algorithms and provides three key features on top of it: an environment model API, a model-based RL agent base class that recovers state-ofthe-art algorithms via subclasses, and an experiment harness to evaluate both model-based and model-free methods (from TF-Agents) under the same experiment protocol to ensure comparability. See Figure 1 for a schematic overview over the framework, a pseudo code snippet for agent assembly and experiment execution, and a list of supported algorithms.

Environment Model
The main idea behind the environment model API is that it mimics a real environment, agents can hence interact with a virtual environment that is learned from data the same way they would interact with the real environment. The environment model comprises individual components for the initial state distribution, state transitions as well as the reward and termination function. At the time of writing, trainable transition models are provided that are neural-network-based and uncertainty-aware, but the software design also enables e.g. Gaussian-process-based transitions and to make the other components trainable.

Agent Classes
The base class for model-based RL contains the environment model components as attributes and a single train method that accepts an extra argument to indicate which component to train. All existing model-based algorithms can be divided into two categories which are covered as subclasses that inherit from our base class design: one for decisiontime planning and one for background planning (Sutton and Barto, 1998), both of which internally assemble the actual environment model object.
Decision-Time Planners solve a virtual planning problem as determined by the current state of the system and the environment model, the outcome of which is a single action to be executed in the real environment. Bellman provides algorithms that first sample action sequences which are executed in the virtual environment, and then choose the best action sequence proportional to the cumulative trajectory reward. The first action of the best action sequence is then executed. This simple yet versatile type of decision-time planning covers state-of-the-art methods for highly non-linear robotics simulation problems. Regarding action-generation, there are two possible scenarios: generating actions either in a state-unconditioned or in a state-conditioned fashion, explained in the following. The former can be achieved e.g. with random shooting (Nagabandi et al., 2018) or with the cross entropy method (Botev et al., 2013), as supported in Bellman through a PETS agent class following Chua et al. (2018). However, the modular design of Bellman also enables, in principle, state-conditioned action proposals with a virtual policy that maps states to actions. One example for the latter is training a model-free off-policy algorithm on real data but only for virtual execution, as in Piche et al. (2019). Another example is to learn a virtual policy through behaviorally cloning the model-based policy, as in Wang and Ba (2019). Note that Piche et al. (2019) leverage a specific off-policy algorithm for their purpose, but Bellman is versatile in the sense that it can support any implemented off-policy algorithm.
Background Planners cover a large set of algorithms and essentially comprise all modelbased RL methods that do planning but are no decision-time planners. In Bellman, however, the term refers to specific algorithms that train model-free algorithms virtually inside of an environment model. Such algorithms are simple yet versatile and recover state-of-the-art approaches for challenging tasks with non-linear transitions. We acknowledge that there are more background planners that are currently not provided in Bellman, e.g. modelbased value expansion (Buckman et al., 2018)

Experiment Harness
Bellman provides a single experiment harness class to conduct experiments with both modelfree and model-based agents alike. The experiment harness relies on a flexible configuration for the task, the experiment protocol and the agent. This enables e.g. to systematically compare Bellman's PETS, ME-TRPO, MBPO and TRPO implementations against PPO, SAC, DDPG and TD3 from TF-Agents. This systematic comparison is ensured because all experiments can share the same task configuration file (that specifies the real environment) and the same experiment protocol configuration file (that specifies the total number of agent-environment interactions and recording of evaluation metrics), but each experiment can use a different agent configuration file (that specifies agent hyperparameters). Further systematic comparison across model-based algorithms alone can be ensured via a separate environment model configuration file that guarantees that all model-based algorithms share the exact same environment model design and training routine-differences in performance can then be attributed to the decision-making component of each algorithm.   M van der Wilk, V Dutordoir, S T John, A Artemev, V Adam, and J Hensman. A framework for interdomain and multioutput Gaussian processes. arXiv, 2020.

A) Bellman Framework C) Bellman Algorithms B) Pseudo Code Snippet
T Wang and J Ba. Exploring model-based planning with policy networks. arXiv, 2019.
Y Wu, E Mansimov, S Liao, R Grosse, and J Ba. Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation. In Advances in Neural Information Processing Systems, 2017.