{'fraction_non_alphanumeric': 0.04204426121115476, 'fraction_numerical': 0.016685465365018862, 'mean_word_length': 4.53184165232358, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
On the model-based stochastic value gradient for continuous reinforcement learning

For over a decade, model-based reinforcement learning has been seen as a way to leverage control-based domain knowledge to improve the sample-efficiency of reinforcement learning agents. While model-based agents are conceptually appealing, their policies tend to lag behind those of model-free agents in terms of final reward, especially in non-trivial environments. In response, researchers have proposed model-based agents with increasingly complex components, from ensembles of probabilistic dynamics models, to heuristics for mitigating model error. In a reversal of this trend, we show that simple model-based agents can be derived from existing ideas that not only match, but outperform state-of-the-art model-free agents in terms of both sample-efficiency and final reward. We find that a model-free soft value estimate for policy evaluation and a model-based stochastic value gradient for policy improvement is an effective combination, achieving state-of-the-art results on a high-dimensional humanoid control task, which most model-based agents are unable to solve. Our findings suggest that model-based policy evaluation deserves closer attention.


Introduction
The task of designing a reinforcement learning (RL) agent that can learn to optimally interact with an unknown environment has wide-reaching applications in, e.g., robotics (Kober et al., 2013;Polydoros and Nalpantidis, 2017), control Kiumarsi et al., 2017), finance (Fischer, 2018), and gaming (Vinyals et al., 2019;Berner et al., 2019;Justesen et al., 2019). Many long-standing challenges remain after decades of research (Sutton and Barto, 2018) and the field is unsettled on a single method for solving classes of tasks. Subtle variations in the settings can significantly impact how agents need to learn, explore, and represent their internal decision process and the external world around them.
Model-based methods explicitly construct a surrogate of the true environment, which can be queried with hypothetical sequences of actions to assess their outcome. In contrast, model-free methods rely entirely on online and historical ground-truth data, and implicitly incorporate the environment dynamics into action value estimates. Historically, the abstractions of model-based methods are more amenable to the incorporation of expert domain knowledge (Todorov et al., 2012), and often find higher reward policies than their model-free counterparts when only a small amount of ground-truth data can be collected (Deisenroth and Rasmussen, 2011;Chua et al., 2018;Janner et al., 2019;Kaiser Figure 1: A model-based SAC-SVG agent learns a near-optimal humanoid gait by using shorthorizon model rollouts. More videos are online at sites.google.com/view/2020-svg. et al., 2019). However model-based methods struggle to match the performance of model-free agents when the latter are allowed unlimited interactions with the environment. While recent work has greatly improved the asymptotic performance of model-based agents, on the most challenging tasks they still often significantly under-perform model-free approaches. While the incorporation of an explicit world model can introduce helpful inductive biases to RL agents, imperfect learned models introduce additional, unwanted biases. An imperfect model may assign high expected reward to catastrophic actions, or it may conjure fantasies of hypothetical states having no correspondence to any realizable agent configuration. Precisely which biases are introduced, and the resulting impact on the agent, depends heavily on how the model is used. In some cases, the model aids action selection online, and in others it augments an existing replay buffer with fictional experience. The reproducibility crisis in reinforcement learning (Islam et al., 2017;Henderson et al., 2018;Engstrom et al., 2020;Sinha et al., 2020) makes evaluating and understanding the tradeoffs difficult.
In this work we seek to answer a key question: "What is the simplest model-based agent that can achieve state-of-the-art results on current benchmark tasks?" We show that if the model is used only for policy improvement, then a simple combination of a stochastic value gradient (SVG) , entropy regularization, soft value estimates (Haarnoja et al., 2018), and a single deterministic dynamics model are sufficient to match and surpass the performance of model-based and model-free methods. This finding is particularly notable because our baselines are substantially stronger than those reported in previous work. We also show that the performance improvements of our approach cannot be attributed to model architecture alone. Finally, we demonstrate that policy evaluation can be significantly more sensitive to model bias than policy improvement, even for short rollouts.

Related work on model-based continuous reinforcement learning
In most model-based methods for continuous control, the dynamics model serves some or all of the following functions: (1) a source of low-bias on-policy value estimates, obtained by explicitly unrolling the policy for the full horizon or by refining a bootstrapped value estimate; (2) a source of multistep value gradients for policy improvement; or (3) a source of fictional transition data with which to augment expensive ground-truth transition data for standard model-free updates. Nearly every combination of these three functions can be  (1) and (2), one obtains a generalized form of PILCO (Deisenroth and Rasmussen, 2011). Introducing bootstrapped value estimates and removing (1), one obtains the value gradient Byravan et al., 2019;Clavera et al., 2020). The model with (1) can also be used in a search procedure (Springenberg et al., 2020;Marino et al., 2020). Conversely if (1) is retained and (2) is removed, the MVE method (Feinberg et al., 2018) is recovered. Dyna-Q (Sutton, 1990) is an early method which uses the model exclusively for (3), followed more recently by NAF (Gu et al., 2016), ME-TRPO (Kurutach et al., 2018), and MBPO (Janner et al., 2019). Other contributions focus on the effect of ensembling some or all of the learned models, such as PETS (Chua et al., 2018), STEVE (Buckman et al., 2018), and POPLIN . We overview the current state of the literature and summarize some key dimensions in table 1. Our paper fits into this space of related work to show that if set up correctly, the value gradient is a competitive baseline.
We have focused this section on continuous spaces and refer to Hamrick et al. (2020) for further discussions and related work in discrete spaces.

Preliminaries, notation, and background in reinforcement learning
Here we present the non-discounted setting for brevity and refer to Thomas (2014); Haarnoja et al. (2018) for the full details behind the γ-discounted setting.

Markov decision processes and reinforcement learning
A Markov decision process (MDP) (Szepesvári, 2010;Puterman, 2014) is a discrete-time stochastic control process widely used in robotics and industrial systems. We represent an MDP as the tuple (X , U, p, r), where X is the state space, U is the control or action space, the transition dynamics p := Pr(x t+1 |x t , u t ) capture the distribution over the next state x t+1 given the current state x t and control u t , r : X × U → R is the reward map. The termination map d : X × U → [0, 1] indicates the probability of the system terminating after executing (x t , u t ). We refer to the dynamics, rewards, and termination map as the world model, e.g. following Ha and Schmidhuber (2018), and consider MDPs with known and unknown world models. We focus on MDPs with continuous and real-valued state spaces X = R m and bounded control spaces U = [−1, 1] n . We consider parameterized stochastic policies π θ (x t ) := Pr(u t |x t ) that induce state and state-control marginals ρ π t (·) for each time step t, which can be constrained to start from an initial state x 0 as ρ π t (·|x 0 ). For finite-horizon non-discounted MDPs, the value V or action-conditional value Q of a policy π is which may be extended to regularize the policy's entropy with some temperature α ≥ 0 as The value function of a given policy can be estimated (i.e. policy evaluation) by explicitly rolling out the world model for H steps with The final value estimatorṼ can take the form of a simple terminal reward function or a parametric function approximator trained through some variant of Bellman backup such as fitted Q-iteration (Antos et al., 2008). Following Feinberg et al. (2018), we refer to eq. (3) as the model-based value expansion (MVE). Policy improvement updates the policy to attain a better expected value. In the RL setting, the world model is often unknown and the value estimate is approximated in a model-free way that does not attempt to explicitly model the world. In many actor-critic methods, policy improvement is done with gradient ascent using the value gradient ∇ θ V π (x). With stochastic policies, ∇ θ V π (x) is the stochastic value gradient (SVG) . For consistency, we refer to methods that update the policy with an H-step value expansion as SVG(H), even though other papers refer to this by other names (Byravan et al., 2019;Clavera et al., 2020).

The soft actor-critic for learning continuous control policies
The soft actor-critic (SAC) method (Haarnoja et al., 2018) learns a state-action value function Q θ , stochastic policy π θ , and a temperature α to find an optimal policy for a continuous-valued MDP (X , U, p, r, γ). SAC optimizes a γ-discounted maximum-entropy objective as in Ziebart et al. (2008); Ziebart (2010); Fox et al. (2015). The policy π θ is a parameterized tanh-Gaussian that given x t , generates samples u t = tanh(µ θ (x t ) + σ θ (x t ) ), where ∼ N (0, 1) and µ θ and σ θ are models that generate the pre-squashed mean and standard deviation. The critic Q θ optimizes the soft (single-step) Bellman residual where D is a distribution of transitions, e.g. an offline replay buffer containing recent experience,θ is an exponential moving average of the weights (Mnih et al., 2015), is the critic target, the soft value function is and log π θ (u|x) is the log-probability of the action. SAC also uses double Q learning (Hasselt, 2010) and does policy optimization with the objective where Q θ,α (x, ·) ∝ exp{ 1 α Q θ,α (x, ·)} and the last equality comes from expanding the KL definition. The temperature α is adapted following Haarnoja et al. (2018) to make the policy's entropy match a target valueH ∈ R by optimizing

SVG(H) with entropy regularization and a model-free value estimate
In this section we discuss a simple model-based extension of model-free SAC. We find that one is immediately suggested by reframing the SAC actor update within the SVG framework.
We also describe our dynamics model architecture, which is better suited for multistep value gradients than the MLPs often employed in model-based RL agents.

Connecting the SAC actor update and stochastic value gradients
Although motivated differently in Haarnoja et al. (2018), the SAC actor update is equivalent to entropy-regularized SVG(0) with a soft value estimate. This is seen by comparing the entropy-regularized value estimate in eq. (2) with the SAC actor objective in eq. (7). Given the empirical success of model-free SAC agents, it is natural to think about a model-based extension. Since the model-free SAC actor update is entropy-regularized SVG(0), simply using the same soft value estimate and entropy adjustment heuristic from SAC, and increasing the SVG horizon from 0 to H immediately provides such an extension. The approach retains some of the desirable characteristics of SAC, such as the effectiveness of the soft value estimate in encouraging exploration, but adds the ability of multi-step SVG to use on-policy value expansions for policy improvement. Algorithm 1 summarizes the approach, which we will refer to SAC-SVG(H), with more details provided in app. A. Briefly put, for policy improvement we minimize the entropy-regularized value estimate by ascending the stochastic value gradient, differentiating eq. (9) with respect to θ. V π,α 0:H is an entropy-regularized value expansion that explicitly unrolls the model for H steps and uses the model-free SAC value estimate at the terminal rollout state-action pair. We adapt the temperature α to make the policy's expected entropy match a target valueH ∈ R by optimizing the J α from SAC in eq. (8).

Approximate world models for deterministic systems
We learn a deterministic transition dynamics model f θ that maps the current state and a sequence of actions to the next sequence of states, i.e. f θ : Our dynamics f θ is autoregressive over time and predicts state deltas with a GRU (Cho et al., 2014) carrying forward a hidden state. We start with the current system state x 1 and an initial hidden state h 1 and use h 1 = 0 in all of our experiments. Given {x t , h t , u t }, we encode the state and action into z t := f enc θ (x t , u t ). We then use a multi-layer GRU to update the hidden state with h t+1 := GRU(z t , h t ), and we then decode the hidden state withx t+1 : . We model f enc θ and f dec θ with multi-layer MLPs. App. A describes more experimental details and our hyper-parameters. For every task we consider, we use a 2-layer MLPs with 512 hidden units and 2-layer GRU.
Learning the dynamics uses multi-step squared error loss where f θ (x 1 , u 1:H−1 ) unrolls the GRU and predicts the next statesx 2:H and the expectation E (x 1:H ,u 1:H−1 )∼D is over H-step sequences uniformly from the replay buffer. We approximate all expectations for the models in this section with a batch size of 1024. We learn the reward map r θ : X × U → R as a neural network with a squared error loss We learn the termination map d θ : X × U → [0, 1] that predicts the probability of the system terminating after executing (x t , u t ), which we observe as d t+1 ∈ {0, 1}. We model d t+1 as a Bernoulli response with likelihood We use a multi-layer neural network to model d θ in the logit space and minimize the negative log-likelihood with the objective We only learn the terminations that are not time-based. Equation (13) could be weighted to deal with an imbalance between termination and non-termination conditions, but in practice we found this weighting to not be important.
Discussion. Our deterministic dynamics model using a GRU for predicting multistep dynamics is simple in comparison to the ensembles of probabilistic models other recent methods use (Chua et al., 2018;Buckman et al., 2018;Janner et al., 2019). The improvements from these models could benefit our base model as well, but our design choice here for simplicity enables us to 1) sample states uniformly from the replay buffer without needing to obtain the most recent hidden state associated with it, 2) still model transitions in a latent space of the GRU, and 3) optimize the multi-step likelihood.  .

Experimental results on MuJoCo locomotion control tasks 1
We evaluate SAC-SVG(H) on all of the MuJoCo (Todorov et al., 2012) locomotion experiments considered by POPLIN, MBPO, and STEVE, all current state-of-the-art related approaches. Note that although we compare against those methods, elements of each could be introduced to SAC-SVG(H) and improve the performance at the cost of increased complexity. We provide a sweep over horizon lengths for the POPLIN tasks and fix H = 2 in the MBPO tasks. For every horizon length, we perform a hyper-parameter search only over the target entropy schedule, which we further describe in app. A. Our SAC baseline uses the same state normalization and target entropy schedule as our SAC-MVE runs in every environment. Table 2 shows the results of our method in comparison to POPLIN (Wang and Ba, 2019) on the locomotion tasks they consider from , POPLIN uses the ground-truth reward for these tasks and learns a model -we learn both. We outperform POPLIN in most of the locomotion tasks, though it has a strong exploration strategy and works exceedingly well in the cheetah environment from PETS (Chua et al., 2018). Notably our SAC baseline often also outperforms the SAC baseline reported in . We are able to find a policy that generates an optimal action with a single rollout sample rather than the thousands of rollouts POPLIN typically uses and find that setting H = 2 usually improves upon our SAC baseline (H = 0). Figure 2 shows our results in comparison to MBPO and STEVE, which evaluate on the MuJoCo tasks in the OpenAI gym (Brockman et al., 2016) that are mostly the standard v2 tasks with early termination and alive bonuses, and with a truncated observation space for the humanoid and ant that discards the inertial measurement units. SAC-SVG(H) consistently matches the best performance and convergence rates across every task and is able to learn a running gait on the humanoid ( fig. 1).
1. Our source code is online at github.com/facebookresearch/svg and builds on the SAC implementation from Yarats and Kostrikov (2020). Videos of our agents are available at sites.google.com/view/2020-svg. We report the mean evaluation rewards and standard deviations across ten trials. The gap between the holdout curve and the test curve is error due to distribution shift between the train and test distributions. Since the ensemble of MLPs generalizes better in the supervised setting, our performance improvement cannot be attributed to a simple change in model architecture.

Model architectures and ensembling
PETS, POPLIN, and MBPO all relied on very similar implementations of bootstrapped ensembles of MLPs to model dynamics. Since we use a different architecture, it is important to consider the impact of the change on the agent's performance. Simply comparing the reward curves resulting from different model choices does not provide any insight into why some methods perform better than others. In particular, such head-to-head comparisons cannot differentiate between a model's overall ability to generalize and more subtle interactions with the rest of the agent, such as the effect on exploration. To explore the difference in generalization ability between different architectures, we removed the dependence of the data collection process on the dynamics model by comparing the architectures on an independent episodic replay buffer, generated by a standard modelfree SAC trial. By sequentially adding episodes to the model's training and holdout datasets, retraining the model, and testing on a fixed test split, we isolated the generalization characteristics of the models while simulating the online nature of the RL data-collection process. The results of this experiment are presented in fig. 3. An ensemble of MLPs generalizes better than a single recurrent network of similar capacity. We selected the recurrent network for its amenability to multistep value gradient backpropagation. While we could ensemble our recurrent architecture, we observe that a single model obtains competitive results, and have chosen to prioritize simplicity. Interestingly, even though the single recurrent model overfits much more heavily to the training data, the asymptotic reward of our humanoid agent is significantly higher and qualitatively different than the agent in Janner et al. (2019). Hence it is not clear how well the model needs to generalize, since short-horizon rollouts are typically initiated with states from the replay buffer (i.e. the model's training data).

Value expansions in the actor and critic
In fig. 4, we consider the effect of introducing the critic MVE update of Feinberg et al. replay buffer, generated a large batch of fictional transitions, and proceeded to repeatedly update the actor/critic models on those stored transitions.
The contrast between our results and those of the MBPO ablation are instructive. Whereas Janner et al. (2019) reported that MVE significantly underperformed MBPO in terms of final reward, even when H = 1, we find that MVE is competitive on all environments until a certain point in learning, after which performance gradually degrades. A likely explanation lies in the MVE dynamics model update. As the replay buffer grows, the dynamics model update is less likely to sample recent transitions in each minibatch, resulting in a gradual increase in model error ( fig. 7). As in fig. 3, the increase in dynamics model error cannot be attributed to the capacity of the model architecture, since the supervised training error on an equivalent replay buffer is significantly lower than the online training error. This observation highlights the impact of model error when the model-generated transitions are used to update the critic, and supports van Hasselt et al. (2019) argument that inaccurate parametric forward dynamics models may be particularly detrimental to value learning.

Conclusion
SAC-SVG(H) combines the most effective ideas from recent MBRL research, resulting in a simple, robust model-based agent. A few key future directions and applications are in: 1. Policy refinement or semi-amortization as in Marino et al. (2020) interprets the policy as solving a model-based control optimization problem and can be combined with differentiable control (Okada et al., 2017;Amos et al., 2018;Pereira et al., 2018;Agrawal et al., 2019;East et al., 2020). Fine-tuning can also be performed at an episode-level as in Nagabandi et al. (2018) to adapt the dynamics to the observations in the episode.

4.
Unsupervised pretraining or self-supervised learning using world models (Ha and Schmidhuber, 2018;Shyam et al., 2018;Pathak et al., 2019;Sharma et al., 2020;Sekar et al., 2020) push against the tabula rasa viewpoint of agents starting from zero knowledge about the environment it is interacting with and would allow them to start with a reasonable idea of what primitive skills the system enables them to do.

5.
Going beyond the single-agent, single-task, online setting. Some of the core ideas behind model-based value expansion can be applied to more sophisticated settings. In multi-agent settings, an agent can consider short-horizon rollouts of the opponents. In the batch RL setting (Fujimoto et al., 2018a;Kumar et al., 2019;Wu et al., 2019) the short-horizon rollouts can be used to constrain the agent to only considering policies that keep the observations close to the data manifold of observed trajectories.
Algorithm 1 Our combination of SAC and SVG(H). Components of SAC are in colored in black and the model-based SVG components are in purple.
Hyperparameters: #updates M step , M seq , target network update τ , planning horizon H Models: Actor π θ , critic ensemble Q θ , temperature α, dynamics f θ , reward r θ , termination d θ Initialize the replay buffer D for environment step t = 1..T do Sample u t ∼ π(x t ) and execute u t on the system to obtain (r t , x t+1 , d t+1 ) and append it to D for M step updates do Sample a batch of single-step transitions θ π ← grad_update(θ π , ∇ θπ J SVG π,α (D step )) Fit the SVG(H) actor with (9) α ← grad_update(α, ∇ α J α (D step )) Update the temperature with (8) Fit the critic ensemble with (4) θ r ← grad_update(θ r , ∇ θr J r (D step )) Fit the reward model with (11) Fit the termination model with (13) Fit the multi-step dynamics model with (10)  We explicitly decay the policy's target entropyH rather than keeping it fixed the entire episode as done in vanilla SAC. The target entropy is important for balancing exploration and exploitation and manually decaying it helps control the agent in data-limited settings. This allows us to start the training with a high-entropy policy that we explicitly anneal down to a lower entropy by the end of training. We do this with the exponential decaȳ where T is the number of training timesteps,H init is the initial target entropy,H final is the final target entropy, and β is the decay factor. We plot an example of this in fig. 6.  Figure 5: Results on the environments tasks considered in POPLIN. We run SAC-SVG for ten trials and report the mean and standard deviation of the reward.

A.2. Hyper-parameters and random search
We share the hyper-parameters in table 3 between the tasks and only search over the horizon and target entropy values, which we show in table 4. We only perform a hyper-parameter search over the target entropy decay rates for each task, which is important to learn as it impacts how the agent explores in the environment. We found SAC-SVG(H) to be more sensitive to the target entropy decay rate and posit it is important to help the policy interact with the model-based components in the earlier phases of training. We perform a random search over 20 seeds for each task to find a target entropy schedule, where we sampleH init ∼ Cat({1, 0, −1, −2}),H final ∼ Cat {H init , −5} ∪ {−2 i |i ∈ {0, . . . , 6}} , and γ ∼ Cat {2 i |i ∈ {0, . . . , 6}} , where Cat(·) is a uniform categorical distribution.

A.3. Walker experiment analysis when doing critic MVE
We provide additional data from selected walker trials that use value expansion on the critic and SVG(H) expansion on the actor behind the summary shown in fig. 4. In MVE trials that perform poorly we can see the model error increase until eventually the agent stops improving. As noted in the previous section, this phenomenon highlights the impact of model error when the model-generated transitions are used to update the critic, and lends credence to the argument of van Hasselt et al. (2019), who also suggest that inaccurate parametric forward dynamics models may be particularly detrimental to value learning.    . We find value-expanding in the actor typically leads to more stable behavior.