{'fraction_non_alphanumeric': 0.04874304672928049, 'fraction_numerical': 0.018520046877837017, 'mean_word_length': 4.306937631394534, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Deep Reinforcement Learning Approach to Marginalized Importance Sampling with the Successor Representation

Marginalized importance sampling (MIS), which measures the density ratio between the state-action occupancy of a target policy and that of a sampling distribution, is a promising approach for off-policy evaluation. However, current state-of-the-art MIS methods rely on complex optimization tricks and succeed mostly on simple toy problems. We bridge the gap between MIS and deep reinforcement learning by observing that the density ratio can be computed from the successor representation of the target policy. The successor representation can be trained through deep reinforcement learning methodology and decouples the reward optimization from the dynamics of the environment, making the resulting algorithm stable and applicable to high-dimensional domains. We evaluate the empirical performance of our approach on a variety of challenging Atari and MuJoCo environments.


Introduction
Off-policy evaluation (OPE) is a reinforcement learning (RL) task where the aim is to measure the performance of a target policy from data collected by a separate behavior policy (Sutton & Barto, 1998). As it can often be difficult or costly to obtain new data, OPE offers an avenue for reusing previously gathered data, making OPE an important challenge for applying RL to real-world domains (Zhao et al., 2009;Mandel et al., 2014;Swaminathan et al., 2017;Gauci et al., 2018).
Marginalized importance sampling (MIS) (Liu et al., 2018;Xie et al., 2019;Nachum et al., 2019a) is a family of OPE methods which re-weight sampled rewards by directly learning the density ratio between the state-action occupancy of the target policy and the sampling distribution. This approach can have significantly lower variance than traditional importance sampling methods (Precup et al., 2001), which consider a product of ratios over trajectories, and is amenable to deterministic policies and behavior agnostic settings where the sampling distribution is unknown. However, the body of MIS work is largely theoretical, and as a result, empirical evaluations of MIS have mostly been carried out on simple low-dimensional tasks, such as mountain car (state dim. of 2) or cartpole (state dim. of 4). In comparison, deep RL algorithms have shown successful behaviors in high-dimensional domains such as Humanoid locomotion (state dim. of 376) and Atari (image-based).
In this paper, we present a straightforward approach for MIS that can be computed from the successor representation (SR) (Dayan, 1993) of the target policy by directly optimizing the reward function. Our algorithm, the Successor Representation DIstribution Correction Estimation (SR-DICE), is the first method that allows MIS to scale to high-dimensional systems, far outperforming previous approaches. In comparison to previous algorithms which rely on minimax optimization or kernel methods (Liu et al., 2018;Nachum et al., 2019a;Uehara & Jiang, 2019;Mousavi et al., 2020;Yang et al., 2020), SR-DICE requires only a simple convex loss applied to a linear function, after computing the SR. Similar to the deep RL methods which can learn in high-dimensional domains, the SR can be computed easily using behavior-agnostic temporal-difference (TD) methods. This makes our algorithm highly amenable to deep learning architectures and applicable to complex tasks.
The SR, which measures the expected future occupancy of states for a given policy, has a clear relationship to MIS methods, which estimate the ratio between the occupancy of state-action pairs and the sampling distribution. However, this relationship is muddied in a deep RL context, where the deep SR measures the expected future sum of feature vectors. Our approach, SR-DICE, provides a straightforward and principled method for extracting density ratios from the SR without any modifications to the standard learning procedure of the SR. Access to these density ratios is valuable as they have a wide range of possible applications such as policy regularization (Nachum et al., 2019b;Touati et al., 2020), imitation learning (Kostrikov et al., 2019), off-policy policy gradients (Imani et al., 2018;Liu et al., 2019b;Zhang et al., 2019), non-uniform sampling procedures (Sinha et al., 2020), or for mitigating distributional shift in offline RL (Fujimoto et al., 2019b;Kumar et al., 2019).
We highlight the value of the MIS density ratios for one reason in particular-in our theoretical analysis we prove that SR-DICE and the deep SR produce exactly the same value estimate. This is surprising as SR-DICE takes a distinct approach for value estimation by re-weighting every reward in the dataset with an importance sampling ratio while the deep SR estimates the value in a similar fashion to TD learning. This theoretical result extends to the deep RL setting and is consistent in our experimental results. This result is a double-edged sword which (negatively) implies there is no discernible difference of using our MIS approach for policy evaluation, but (positively) implies the estimated density ratios are accurate enough to match the performance of TD methods. This is an important observation as our empirical results demonstrate that previous MIS methods scale very poorly in comparison to TD methods to high dimensions, which is consistent with prior results (Voloshin et al., 2019;Fu et al., 2021). Even if there is no difference for OPE, a MIS method which matches the performance of TD-based methods is desirable if we are concerned with estimating the density ratios of the target policy.
We benchmark the performance of SR-DICE on several high-dimensional domains in MuJoCo (Todorov et al., 2012) and Atari (Bellemare et al., 2013), against several recent MIS methods (Nachum et al., 2019a;Zhang et al., 2020a). Our results demonstrate several key findings regarding highdimensional tasks.
Current MIS methods underperform deep RL at highdimensional tasks. While previous results have shown that MIS methods can produce competitive results to TD methods, our empirical results show that MIS methods scale poorly to challenging tasks. In Atari we find that the baseline MIS method exhibit unstable estimates, often reaching errors with many orders of magnitude. Comparatively, the baseline deep RL methods, which rely on TD learning and have a history of achieving high performances in the control setting (Mnih et al., 2015;Schulman et al., 2017;Fujimoto et al., 2018), outperform the MIS baselines at every task and often by a wide margin.
SR-DICE outperforms current MIS methods at policy evaluation and therefore density ratio estimation. Our empirical results confirm our theoretical analysis, which state that SR-DICE and the standard deep SR approach should produce identical value estimates (with differences due only to changes to the optimization process). While this result may initially sound discouraging, given the direct SR approach is comparable to TD learning, and TD learning significantly outperforms current MIS methods, this result also implies that SR-DICE is a much stronger technique for estimating density ratios than previous methods.
Ultimately, while SR-DICE produces a similar result to existing deep RL approaches for policy evaluation, it does provide a practical, scalable, and state-of-the-art approach for estimating state-action occupancy density ratios, while highlighting connections between the SR, reward function optimization, and state-action occupancy estimation. For ease of use and reproduction, our code is open-sourced (https://github.com/sfujim/SR-DICE).

Background
Reinforcement Learning. RL is a framework for maximizing accumulated reward of an agent interacting with its environment (Sutton & Barto, 1998). This problem is typically framed as a Markov Decision Process (MDP) (S, A, R, p, d 0 , γ), with state space S, action space A, reward function R, dynamics model p, initial state distribution d 0 and discount factor γ. An agent selects actions according to a policy π : S × A → [0, 1]. In this paper we address the problem of off-policy evaluation (OPE) problem where the aim is to measure the normalized expected per-step reward of the policy R(π) = (1 − γ)E π [ ∞ t=0 γ t r(s t , a t )]. An important notion in OPE is the value function Q π (s, a) = E π [ ∞ t=0 γ t r(s t , a t )|s 0 = s, a 0 = a], which measures the expected sum of discounted rewards when following π, starting from the state-action pair (s, a).
We define d π (s, a) as the discounted state-action occupancy, the probability of seeing (s, a) under policy π with discount γ: d π (s, a) = (1 − γ) ∞ t=0 γ t s0 d 0 (s 0 )p π (s 0 → s, t)π(a|s)ds 0 , where p π (s 0 → s, t) is the probability of arriving at the state s after t time steps when starting from an initial state s 0 . This distribution is important as R(π) equals the expected reward r(s, a) under d π : R(π) = E (s,a)∼d π ,r(s,a) [r(s, a)]. (1) A common approach for estimating R(π) is through temporal-difference (TD) learning (Sutton, 1988) where an estimate of the value function Q(s, a) is updated over individual transitions (s, a, r(s, a), s ) by the following: where a is sampled according to the target policy π and α is the learning rate. Provided an infinite set of transitions, TD learning is known to converge to the true value function in the off-policy setting (Jaakkola et al., 1994;Sutton & Barto, 1998). TD learning can also be applied to other learning problems, such as the successor representation, where the reward r(s, a) in Equation (2) is replaced with the quantity of interest.
Successor Representation. The successor representation (SR) (Dayan, 1993) of a policy is a measure of occupancy of future states. It can be viewed as a general value function that learns a vector of the expected discounted visitation for each state. The SR Ψ π of a given policy π is defined as Ψ π (s |s) = E π [ ∞ t=0 γ t 1(s t = s )|s 0 = s]. Importantly, the value function can be recovered from the SR by summing over the expected reward of each state V π (s) = s Ψ π (s |s)E a ∼π [r(s , a )]. For infinite state and action spaces, the SR can instead be generalized to the expected occupancy over features, known as the deep SR (Kulkarni et al., 2016) or successor features (Barreto et al., 2017). For a given encoding function φ : S×A → R n , the deep SR ψ π : S × A → R n is defined as the expected discounted sum of features from the encoding function φ when following the policy from a given state-action pair: If the encoding φ(s, a) is learned such that the original reward function is a linear function of the encoding r(s, a) = w φ(s, a), then similar to the original formulation of SR, the value function can be recovered from a linear function of the SR: Q π (s, a) = w ψ π (s, a). The deep SR network ψ π is trained to minimize the MSE between ψ π (s, a) and φ(s, a) + γψ (s , a ) on transitions (s, a, s ) sampled from the dataset. A frozen target network ψ is used to provide stability (Mnih et al., 2015;Kulkarni et al., 2016), and is updated to the current network ψ ← ψ π after a fixed number of time steps. The encoding function φ is typically trained by an encoder-decoder network (Kulkarni et al., 2016;Machado et al., 2017;2018a). For OPE where the reward function is learned by minimizing w φ(s, a) − r(s, a) 2 , the SR is comparable to TD learning, as they both estimate the discounted sum of future rewards and use similar updates.
Marginalized Importance Sampling. Marginalized importance sampling (MIS) is a family of importance sampling approaches for off-policy evaluation in which the performance R(π) is evaluated by re-weighting rewards sampled from a dataset D = {(s, a, r, s )} ∼ p(s |s, a)d D (s, a), where d D is an arbitrary distribution, typically but not necessarily, induced by some behavior policy. It follows that R(π) can computed with importance sampling weights on the rewards d π (s,a) d D (s,a) : The goal of marginalized importance sampling methods is to learn the weights w(s, a) ≈ d π (s,a) d D (s,a) , using data contained in D. The main benefit of MIS is that unlike traditional importance methods, the ratios are applied to individual transitions rather than complete trajectories, which can reduce the variance of long or infinite horizon problems. In other cases, the ratios themselves can be used for a variety of applications which require estimating the occupancy of state-action pairs.

A Reward Function Perspective on Distribution Corrections
In this section, we present our behavior-agnostic approach to estimating MIS ratios, called the Successor Representation DIstribution Correction Estimation (SR-DICE). Our main insight is that MIS can be viewed as an optimization over a learned reward function, where the loss is uniquely optimized when the virtual reward is the MIS density ratio.
Our derived loss function is a straightforward convex loss over the learned reward and the corresponding value function of the target policy. This naturally suggests the use of the successor representation which allows us to maintain an estimate of the value estimate while directly optimizing the reward function. This disentangles the learning process, where the propagation of reward through the MDP can be learned separately from the optimization of the reward. In other words, rather than learn a reward function and value function simultaneously, we tackle each separately, changing the difficult minimax optimization of previous methods into two phases. Interestingly enough, we show that our MIS estimator produces the identical value estimate as traditional deep SR methods. This means the challenging aspect of learning has been pushed onto the computation of the SR, rather than optimizing the density ratio estimate. Fortunately, we can leverage deep RL approaches (Mnih et al., 2015;Kulkarni et al., 2016) to make learning the SR stable, giving rise to a practical MIS method for high-dimensional tasks.
This section begins with the derivation of our core ideas, which shows MIS ratios can be learned through reward function optimization. We then highlight how the SR can be used for reward function optimization in the tabular domain. Finally, we generalize our results to the deep SR setting.

Basic Derivation
In MIS, our aim is to determine the MIS ratios d π (s,a) d D (s,a) , using only data sampled from the dataset D and the target policy π. This presents a challenge as we have direct access to neither d π nor d D .
As a starting point, we begin by following the derivation of DualDICE (Nachum et al., 2019a). We first consider the convex function 1 2 mx 2 − nx, which is uniquely minimized by x * = n m . Now by replacing x with a virtual reward r(s, a), m with the density of the dataset d D (s, a), and n with the density of the target policy d π (s, a), we have reformulated the convex function as the following: As Equation (5) is still the convex function with renamed variables, following Nachum et al. (2019a), we can observe the following: Observation 1 The objective J(r) is minimized when r(s, a) = d π (s,a) d D (s,a) for all state-action pairs (s, a). Equation (5) is an optimization over two expectations over d D and d π . While the first expectation over d D is tractable by sampling directly from the dataset D, the second expectation relies on the state-action visitation of the target policy d π (s, a) which is not directly accessible without a model of the MDP. At this point, we highlight our choice of notation,r(s, a), in Equation (5). Describing the objective in terms of a fictitious rewardr will allow us to draw on familiar relationships between rewards and value functions. Consider the equivalence between the value function over initial state-action pairs (s 0 , a 0 ) and the expectation of rewards over the state-action visitation of the policy . It follows that the expectation over d π in Equation (5) can be replaced with a value functionQ π overr: min r(s,a)∀(s,a) In other words, by noting that the value function is simply the (scaled) expected reward when sampled from the state-action visitation of the target policy, we can replace the impractical expectation over d π with a tractable value function. This form of the objective, Equation (6), is convenient because we can estimate the expectation over d D by sampling directly from the dataset andQ π can be computed using any policy evaluation method.
While we can estimate both terms in Equation (6) with relative ease, the optimization problem is not directly differentiable and would require re-learning the value functionQ π with every adjustment to the learned rewardr. Fortunately, there exists a straightforward paradigm which enables direct reward function optimization known as successor representation (SR).

Tabular SR-DICE
We will begin by discussing how we can apply the SR to MIS in the tabular setting and then generalize our method to non-linear function approximation afterwards. Consider the relationship between the SR Ψ π of the target policy π and its value function: It follows that we can create an optimization problem directly over the reward functionr by modifying Equation (6) to use the SR: Since this optimization problem is convex, it has a closed form solution. The unique optimizer of Equation (8) is: Theorem 1 Equation (9) is the optimal solution to Equation (8) and is equal to d π (s,a) d D (s,a) .
A direct consequence of this result is that Equation (9) can be used with MIS policy evaluation to return the true value estimate 1 |D| (s,a)∈D d π (s,a) d D (s,a) r(s, a) = R(π). Unfortunately, the form of Equation (9) relies on the true SR Ψ π , as well as an expectation over s 0 , both of which may be unobtainable in the setting where we are sampling from a finite dataset D. However, we can still show that with an inexact SRΨ and sampled estimate of the expectation, using the set of start states D 0 in the dataset, approximating the optimizer Equation (9) with gives an MIS estimator 1 |D| (s,a)∈D r * (s, a)r(s, a) of R(π) which is identical to the estimate of R(π) computed directly with the SR.
Theorem 2 Letr(s, a) be the average reward in the dataset D at the state-action pair (s, a).
The take-away is that even when estimating the SR, the approximate density ratio defined by r * is of sufficiently high quality to match the performance of directly estimating the value with the SR.

SR-DICE
Now we will consider how this MIS estimator can be generalized to continuous states by considering the deep SR ψ π over features φ(s, a) and optimizing the weights of a linear function w.
SR Refresher. We begin with a reminder of the details of the deep SR algorithm. The deep SR measures the expected sum of features ψ π (s, a If the reward can be defined as a linear function over the features r(s, a) = w φ(s, a) then the value function can be recovered via a linear function over the deep SR Q(s, a) = w ψ π (s, a). The typical deep SR pipeline follows three steps: 1. Learn the encoding φ. 2. Learn the deep SR ψ π over the encoding φ. 3. Learn w SR by minimizing w SR φ(s, a) − r(s, a) 2 .
We leave the first two stages vague as there is flexibility in how they are approached. This most commonly involves training the encoding φ via an encoder-decoder network to reconstruct transitions and training the deep SR ψ π using TD learning-style methods (Kulkarni et al., 2016;Machado et al., 2018a). While we follow this standard practice, specific details are unimportant for our analysis and we relegate implementation-level details to the appendix.
Given the deep SR ψ π , we can use it to learn the MIS ratio. Recall our objective of reward function optimization (Equation (6)). In the deep SR paradigm, both the reward and value function are determined by linear functions with respect to a single weight vector w. Consequently, we can modify Equation (6) with these linear functions and then optimize the linear weights w directly: where in practice we replace the expectations with samples from the dataset D and the subset of start states D 0 : Algorithm 1 SR-DICE Input: SR ψ, target network ψ , encoder φ, decoder D.

(12)
Again, since the optimization problem Equation (12) is still convex, it has a closed form solution. Let Φ be a |D| × F matrix where each row is the feature vector φ(s, a) with F features. Let Ψ be a |D 0 ||A| × F matrix where each row is the SR weighted by its probability under the policy π(a 0 |s 0 )ψ π (s 0 , a 0 ). Let 1 be a |D 0 ||A| dimensional vector of all 1. The unique optimizer w * of Equation (12) is a F dimensional vector defined as follows: In practice, a matrix-based solution is often undesirable and we may prefer iterative, gradient-based solutions for scalability. In this case, we can directly minimize Equation (12) by taking gradient steps with respect to w.
We now introduce our algorithm Successor Representation stationary DIstribution Correction Estimation (SR-DICE). SR-DICE follows the same first two steps of the standard SR procedure, but replaces the third step with optimizing Equation (12). Given w, an estimate of R(π) can be returned by 1 |D| (s,a,r(s,a))∈D w φ(s, a)r(s, a), where w φ(s, a) ≈ d π (s,a) d D (s,a) . We summarize SR-DICE in Algorithm 1.
We now remark upon two important properties of SR-DICE. The first concerns the quality of the quality of the learned MIS ratio. Although it is difficult to make any guarantees on the accuracy of an approximate ψ π trained with deep RL techniques, if we assume ψ π is exact, then we can show that SR-DICE learns the least squares estimator to the desired density ratio.
Theorem 3 If the deep SR is exact, such that (1 − γ)E s0,a0 [ψ π (s 0 , a 0 )] = E (s,a)∼d π [φ(s, a)], and the sup-port of d π is contained in the dataset D, then the optimizer w * of Equation (12), as defined by Equation (13), is the least squares estimator of (s,a)∈D w φ(s, a) − d π (s,a) The take-away from Theorem 3 is that our optimization problem, at least in the idealized setting, produces the same density ratios as directly learning them. This also means that the main source of error in SR-DICE is in the first two phases: learning the encoding φ and the deep SR ψ π . Notably, both of these steps are independent of the main optimization problem of learning w, as we have shifted the challenging aspects of density ratio estimation onto learning the deep SR. This leaves deep RL to do the heavy lifting. The remaining optimization problem, Equation (11), only involves directly updating the weights of a linear function, and unlike many other MIS methods, requires no tricky minimax optimization.
The second important property of SR-DICE is that Theorem 2 can be extended to the deep SR setting. That is, when derived from the same approximate SR, the optimal solution to both the SR-DICE estimator and the direct SR estimator produce identical estimates of R(π).
Theorem 4 Given the least squares estimator w SR of (s,a)∈D w φ(s, a) − r(s, a) 2 and the optimizer w * of Equation (12), as defined by Equation (13), then the traditional SR estimator This means that SR-DICE produces the same value estimate as the traditional deep SR algorithms, up to errors in the optimization process of w. In other words, SR-DICE does not suffer from the same instability issues that plague other MIS methods when tackling high-dimensional domains where deep RL methods excel (relative to more traditional methods). Although, we typically think of the objective of MIS methods as policy evaluation, since SR-DICE and traditional deep SR produce the same value estimate, there is not a strong argument for using SR-DICE for policy evaluation. However, this also suggests that the estimated density ratios are of reasonably high quality since SR-DICE achieves the same performance as deep RL approaches. Therefore, we can treat SR-DICE is a tractable method for accessing the state-action occupancy of the target policy.

Related Work
Off-Policy Evaluation. Off-policy evaluation (OPE) is a well-studied problem with several families of approaches. One family of approaches is based on importance sampling, which re-weights trajectories by the ratio of likelihoods under the target and behavior policy (Precup et al., 2001). Importance sampling methods are unbiased but suffer from variance which can grow exponentially with the length of trajectories (Li et al., 2015;Jiang & Li, 2016). Consequently, research has focused on variance reduction (Thomas & Brunskill, 2016;Munos et al., 2016;Farajtabar et al., 2018)

Experiments
To evaluate our method, we perform several off-policy evaluation (OPE) experiments on a variety of domains. The aim is to evaluate the normalized average discounted reward E (s,a)∼d π ,r [r(s, a)] of a target policy π. We benchmark our Continuous Action Experiments. We evaluate the methods on a variety of MuJoCo environments (Brockman et al., 2016;Todorov et al., 2012). We examine two experimental settings. In both settings the target policy π and behavior policy π b are stochastic versions of a deterministic policy π d obtained from training the TD3 algorithm (Fujimoto et al., 2018). We evaluate a target policy π = π d + N (0, σ 2 ), where σ = 0.1.
• For the easy setting, we gather a dataset of 500k transi-tions using a behavior policy π b = π d + N (0, σ 2 b ), where σ b = 0.133. This setting roughly matches the experimental setting used by GradientDICE Zhang et al. (2020a). • For the hard setting, we gather a significantly smaller dataset of 50k transitions using a behavior policy which acts randomly with p = 0.2 and uses π d + N (0, Unless specified otherwise, we use a discount factor of γ = 0.99 and all hyper-parameters are kept constant across environments. All experiments are performed over 10 seeds. We display the results of the easy setting in Figure 1 and the hard setting in Figure 2. Atari Experiments. To demonstrate our approach can scale to even more complex domains, we perform experiments with several Atari games (Bellemare et al., 2013), which are challenging due to their high-dimensional image-based state space. Standard pre-processing steps are applied (Castro et al., 2018) and sticky actions are used (Machado et al., 2018b) to increase difficulty and remove determinism. Each method is trained on a dataset of one million time steps. The target policy is the deterministic greedy policy trained by Double DQN (Van Hasselt et al., 2016). The behavior policy is the -greedy policy with = 0.1. We use a discount factor of γ = 0.99. Experiments are performed over 3 seeds. Results are displayed in Figure 3. Additional experiments with different behavior policies can be found in the appendix.
Evaluating the MIS ratios. To evaluate the quality of the MIS ratios themselves, we perform a randomized reward experiment. As the MIS ratio is only the value w that will return the true value of R(π) = E D [w · r(s, a)] for all possible reward functions (Uehara & Jiang, 2019), we generate a large set of rewards functions with a randomlyinitialized neural network, and evaluate the estimate of R(π) obtained from each MIS method on each reward function. The ground-truth is estimated by a set of 100 on-policy trajectories generated by π. We generate 1000 reward functions, with scalar values in the range [0, 10] and remove any redundant reward functions from the set. The MIS ratios and dataset are taken from the hard setting. Experiments are performed over 5 seeds. We report the results in Figure 4.
Discussion. Across the board we find SR-DICE significantly outperforms the MIS methods. Looking at the estimated values of R(π) in the continuous action environments, Figure 2, we can see that SR-DICE converges rapidly and maintains a stable estimate, while the MIS methods are particularly unstable, especially in the case of DualDICE. These observations are consistent in the Atari domain (Figure 3). In accordance with our theoretical analysis, Deep SR and SR-DICE perform similarly in every task, further suggesting that the limiting factor in SR-DICE is the quality of the deep successor representation, rather than learning the density ratios. In the randomized reward experiment, we find that SR-DICE vastly outperforms the other MIS methods in average log MSE, and compares favorably against the other MIS methods in over 80% of reward functions. In the most challenging task, Humanoid, SR-DICE is the best method in over 98% of reward functions. This suggests that SR-DICE provides much higher quality MIS ratio estimates than previous methods.
Ablation. To study the robustness of SR-DICE relative to the competing methods, we perform an ablation study and investigate the effects of dataset size, discount factor, and two different behavior policies. Unless specified otherwise, we use experimental settings matching the hard setting. We report the results in Figure 5. In the dataset size experiment (a), SR-DICE perform well with as few as 5k transitions (5 trajectories). In some instances, the performance is unexpectedly improved with less data, although incrementally. For small datasets, the SR methods outperform Deep TD. One hypothesis is that the encoding acts as an auxiliary reward and helps stabilize learning in the low data regime. In (b) we report the performance over changes in discount factor. The relative ordering across methods is unchanged. In (c) we use a behavior policy of N (0, σ 2 b ), with σ b = 0.5, a much larger standard deviation than either setting for continuous control. The results are similar to the original setting, with an increased bias on the deep RL methods. In (d) we use the underlying deterministic policy as both the behavior and target policy. Even though this setup should be easier since the task is no longer off-policy, the baseline MIS methods perform surprisingly poorly, once again demonstrating their weakness on high-dimensional domains.

Conclusion
In this paper, we introduce a method which can perform marginalized importance sampling (MIS) using the successor representation (SR) of the target policy. This is achieved by deriving an MIS formulation that can be viewed as reward function optimization. By using the SR, we effectively disentangle the dynamics of the environment from learning the reward function. This allows us to (a) use well-known deep RL methods to effectively learn the SR in challenging domains (Mnih et al., 2015;Kulkarni et al., 2016) and (b) provide a straightforward loss function to learn the density ratios without any optimization tricks necessary for previous methods (Liu et al., 2018;Uehara & Jiang, 2019;Nachum et al., 2019a;Zhang et al., 2020c;Yang et al., 2020). Our resulting algorithm, SR-DICE, outperforms prior MIS methods in terms of both performance and stability and is the first MIS method which demonstrably scales to highdimensional problems. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Proof.
Take the partial derivative of J(r) with respect tor(s, a): Then setting ∂J(r) ∂r(s,a) = 0, we have that J(r) is minimized whenr(s, a) = d π (s,a) d D (s,a) for all state-action pairs (s, a).

A.2. Theorem 1
Theorem 1 Equation (18) is the optimal solution to Equation (17) and is equal to d π (s,a) d D (s,a) .
First we consider the relationship between Equation (17) and Equation (18). Take the gradient of Equation (17): Where we can replace d D (s, a) with 1 |D| (s ,a )∈D 1(s = s, a = a): Setting the above gradient equal to 0 and solving forr(s, a) we have the optimizer of J Ψ (r).
Now consider the relationship between Equation (18) and d π (s,a) d D (s,a) . Recall the definition of the state occupancy d π (s, a): γ t p π (s 0 → s, t)π(a|s) .
Now consider the SR: It follows that the SR and the state occupancy share the relationship: Finally recall that d D (s, a) = 1 |D| (s ,a )∈D 1(s = s, a = a). Note in this case, the relationship is exact and does not rely on an expectation. It follows that: Recall by definition V π (s) = s Ψ π (s |s)E a ∼π [r(s , a )]. It follows that s Ψ π (s |s) a ∈A π(a |s )r(s , a ) is an unbiased estimator of V π (s). It follows that estimating R(π) = (1 − γ)E s0 [V π (s 0 )] with the SR would be computed by Now consider the SR-DICE estimator for R(π). By expanding and simplifying we arrive at the SR estimator for R(π): 1 |D| π(a 0 |s 0 )w ψ π (s 0 , a 0 ). Let: • Φ be a |D| × F matrix where each row is the feature vector φ(s, a) with F features.
• Ψ be a |D 0 ||A| × F matrix where each row is π(a 0 |s 0 )ψ π (s 0 , a 0 ), the SR weighted by its probability under the policy over all states s 0 in dataset of start states D 0 and all actions. • 1 be a |D 0 ||A| dimensional vector of all 1.
We can reformulate Equation (35) as the following: Now take the gradient: And set it equal to 0 to solve for w * : A.5. Theorem 3 a)], and the support of d π is contained in the dataset D, then the optimizer w * of Equation (43), as defined by Equation (44), is the least squares estimator of (s,a)∈D w φ(s, a) − d π (s,a) π(a 0 |s 0 )w ψ π (s 0 , a 0 ).
( 43) and (as derived in Subsection A.4) Proof. Let: • Φ be a |D| × F matrix where each row is the feature vector φ(s, a) with F features.
• Ψ be a |D 0 ||A| × F matrix where each row is π(a 0 |s 0 )ψ π (s 0 , a 0 ), the SR weighted by its probability under the policy over all states s 0 in dataset of start states D 0 and all actions. • 1 x be a x dimensional vector of all 1.
• d π and d D be diagonal |D| × |D| matrices where the diagonal entries contain d π (s, a) and d D (s, a) respectively, for each state-action pair (s, a) in the dataset D.
First note the least squares estimator of (s,a)∈D w φ(s, a) − d π (s,a) By our assumption on the deep SR, we have that: and therefore: It follows that we can simplify w * : A.6. Theorem 4 Theorem 4 Given the least squares estimator w SR of (s,a)∈D w φ(s, a) − r(s, a) 2 and the optimizer w * of Equation (52), as defined by Equation (53), then the traditional SR estimator 1 |D0| s0∈D0 w SR ψ π (s 0 , a 0 ) of R(π) is identical to the SR-DICE estimator 1 |D| (s,a,r(s,a))∈D w * φ(s, a)r(s, a) of R(π).
( 52) and (as derived in Subsection A.4) Proof. Let: • Φ be a |D| × F matrix where each row is the feature vector φ(s, a) with F features.
• Ψ be a |D 0 ||A| × F matrix where each row is π(a 0 |s 0 )ψ π (s 0 , a 0 ), the SR weighted by its probability under the policy over all states s 0 in dataset of start states D 0 and all actions. • 1 be a |D 0 ||A| dimensional vector of all 1.
• R be the D dimensional vector of each reward r(s, a) in the dataset D.
First note the least squares solution for direct SR, where w SR is optimized to reduce the mean squared error between w SR φ(s, a) and r(s, a): It follows that the direct SR solution to R(π) is: Now consider the SR-DICE solution to R(π):

B. Additional Experiments
In this section, we include additional experiments and visualizations, covering extra domains, additional ablation studies, run time experiments and additional behavior policies in the Atari domain.

B.1. Extra Continuous Domains
Although our focus is on high-dimensional domains, the environments, Pendulum and Reacher, have appeared in several related MIS papers (Nachum et al., 2019a;Zhang et al., 2020a). Therefore, we have included results for these domains in Figure 6. All experimental settings match the experiments in the main body, and are described fully in Appendix E.  Figure 6: Off-policy evaluation results for Pendulum and Reacher. The shaded area captures one standard deviation across 10 trials. Even on these easier environment, we find that SR-DICE outperforms the baseline MIS methods.

B.2. Representation Learning & MIS
SR-DICE relies a disentangled representation learning phase where an encoding φ is learned, followed by the deep successor representation ψ π which are used with a linear vector w to estimate the density ratios. In this section we perform some experiments which attempt to evaluate the importance of representation learning by comparing their influence on the baseline MIS methods.
Alternate representations. We examine both DualDICE (Nachum et al., 2019a) and GradientDICE (Zhang et al., 2020c) under four settings where we pass the representations φ and ψ π to their networks, where both φ and ψ π are learned in identical fashion to SR-DICE.
See Appendix D for specific details on the baselines. We report the results in Figure 7. For GradientDICE, no benefit is provided by varying the representations, although using the encoding φ matches the performance of vanilla GradientDICE regardless of the choice of network, providing some validation that φ is a reasonable encoding. Interestingly, for DualDICE, we see performance gains from using the SR ψ π as a representation: slightly as input, but significantly when used with linear networks. On the other hand, as GradientDICE performs much worse with the SR, it is clear that the SR cannot be used as a representation without some degree of forethought. The shaded area captures one standard deviation across 10 trials. We see that using the SR ψ π as a representation improves the performance of DualDICE. On the other hand, GradientDICE performs much worse when using the SR, suggesting it cannot be used naively to improve MIS methods.
Increased capacity. As SR-DICE uses a linear function on top of a representation trained with the same capacity as the networks in DualDICE and GradientDICE, our next experiment examines if this additional capacity provides benefit to the baseline methods. To do, we expand each network in both baselines by adding an additional hidden layer. The results are reported in Figure 8. We find there is a very slight decrease in performance when using the larger capacity networks. This suggests the performance gap from SR-DICE over the baseline methods has little to do with model size.  Figure 8: Off-policy evaluation results on HalfCheetah evaluating the performance benefits from larger network capacity on the baseline MIS methods. "Big" refers to the models with an additional hidden layer. The experimental setting corresponds to the hard setting from the main body. The shaded area captures one standard deviation across 10 trials. We find that there is no clear performance benefit from increasing network capacity.

B.3. Toy Domains
We additional test the MIS algorithms on a toy random-walk experiment with varying feature representations, based on a domain from (Sutton et al., 2009).
Domain. The domain is a simple 5-state MDP (x 1 , x 2 , x 3 , x 4 , x 5 ) with two actions (a 0 , a 1 ), where action a 0 induces the transition x i → x i−1 and action a 1 induces the transition x i → x i+1 , with the state x 1 looping to itself with action a 0 and x 5 looping to itself with action a 5 . Episodes begin in the state x 1 .
Target. We evaluate policy π which selects actions uniformly, i.e. π(a 0 |x i ) = π(a 1 |x i ) = 0.5 for all states x i . Our dataset D contains all 10 possible state-action pairs and is sampled uniformly. We use a discount factor of γ = 0.99. Methods are evaluated on the average MSE between their estimate of d π d D on all state-action pairs and the ground-truth value, which is calculated analytically.
Hyper-parameters. Since we are mainly interested in a function approximation setting, each method uses a small neural network with two hidden layers of 32, followed by tanh activation functions. All networks used stochastic gradient descent with a learning rate α tuned for each method out of {1, 0.5, 0.1, 0.05, 0.01, 0.001}. This resulted in α = 0.05 for DualDICE, α = 0.1 for GradientDICE, and α = 0.05 for SR-DICE. Although there are a small number of possible data points, we use a batch size of 128 to resemble the regular training procedure. As recommended by the authors we use λ = 1 for GradientDICE (Zhang et al., 2020c), which was not tuned. For SR-DICE, we update the target network at every time step τ = 1, which was not tuned.
Since there are only 10 possible state-action pairs, we use the closed form solution for the vector w. Additionally, we skip the state representation phase of SR-DICE, instead learning the SR ψ π over the given representation of each state, such that the encoding φ = x. This allows us to test SR-DICE to a variety of representations rather than using a learned encoding. Consequently, with these choices, SR-DICE has no pre-training phase, and therefore, unlike every other graph in this paper, we report the results as the SR is trained, rather than as the vector w is trained.
Features. To test the robustness of each method we examine three versions of the toy domain, each using a different feature representation over the same 5-state MDP. These feature sets are again taken from (Sutton et al., 2009).
• Tabular features: states are represented by a one-hot encoding, for example x 2 = [0, 1, 0, 0, 0]. • Inverted features: states are represented by the inverse of a one-hot encoding, for example x 2 = 1 2 , 0, 1 2 , 1 2 , 1 2 . • Dependent features: states are represented by 3 features which is not sufficient to cover all states exactly. In this case Since our experiments use neural networks rather than linear functions, this representation is mainly meant to test SR-DICE, where we skip the state representation phase for SR-DICE and use the encoding φ = x, limiting the representation of the SR. Results. We report the results in Figure 9. We remark on several observations. SR-DICE learns significantly faster than the baseline methods, likely due to its use of temporal difference methods in the SR update, rather than using an update similar to residual learning, which is notoriously slow (Baird, 1995;Zhang et al., 2020b). GradientDICE appears to still be improving, although we limit training at 50k time steps, which we feel is sufficient given the domain is deterministic and only has 5 states. Notably, GradientDICE also uses a higher learning rate than SR-DICE and DualDICE. We also find the final performance of SR-DICE is much better than DualDICE and GradientDICE in the domains where the feature representation is not particularly destructive, highlighting the easier optimization of SR-DICE. In the case of the dependent features, we find DualDICE outperforms SR-DICE after sufficient updates. However, we remark that this concern could likely be resolved by learning the features and that SR-DICE still outperforms GradientDICE. Overall, we believe these results demonstrate that SR-DICE's strong empirical performance is consistent across simpler domains as well as the high-dimensional domains we examine in the main body.

B.4. Run Time Experiments
In this section, we evaluate the run time of each algorithm used in our experiments. Although SR-DICE relies on pre-training the deep successor representation before learning the density ratios, we find each marginalized importance sampling (MIS) method uses a similar amount of compute, due to the reduced cost of training w after the pre-training phase.
We evaluate the run time on the HalfCheetah environment in MuJoCo (Todorov et al., 2012) and OpenAI gym (Brockman et al., 2016). As in the main set of experiments, each method is trained for 250k time steps. Additionally, SR-DICE and Deep SR train the encoder-decoder for 30k time steps and the deep successor representation for 100k time steps before training w. Run time is averaged over 3 seeds. All time-based experiments are run on a single GeForce GTX 1080 GPU and a Intel Core i7-6700K CPU. Results are reported in Figure 10. We find the MIS algorithms run in a comparable time, regardless of the pre-training step involved in SR-DICE. This can be explained as training w in SR-DICE involves significantly less compute than DualDICE and GradientDICE which update multiple networks. On the other hand, the deep reinforcement learning approaches run in about half the time of SR-DICE.

B.5. Atari Experiments
To better evaluate the algorithms in the Atari domain, we run two additional experiments where we swap the behavior policy. We observe similar trends as the experiments in the main body of the paper. In both experiments we keep all other settings fixed. Notably, we continue to use the same target policy, corresponding to the greedy policy trained by Double DQN (Van Hasselt et al., 2016), the same discount factor γ = 0.99, and the same dataset size of 1 million.
Increased noise. In our first experiment, we attempt to increase the randomness of the behavior policy. As this can cause destructive behavior in the performance of the agent, we adopt an episode-dependent policy which selects between the noisy policy or the deterministic greedy policy at the beginning of each episode. This is motivated by the offline deep reinforcement learning experiments from (Fujimoto et al., 2019a). As a result, we use an -greedy policy with p = 0.8 and the deterministic greedy policy (the target policy) with p = 0.2. is set to 0.2, rather than 0.1 as in the experiments in the main body of the paper. Results are reported in Figure 11. We observe very similar trends to the original set of experiments. Again, we note DualDICE and GradientDICE perform very poorly, while SR-DICE, Deep SR, and Deep TD achieve a reasonable, but biased, performance. In this setting, we still find the behavior policy is the closest estimate of the true value of R(π) .
Separate behavior policy. In this experiment, we use a behavior which is distinct from the target policy, rather than simply adding noise. This behavior policy is derived from an agent trained with prioritized experience replay and Double DQN (Schaul et al., 2016;Fujimoto et al., 2020). Again, we use a -greedy policy, with = 0.1. We report the results in Figure 12.  Figure 12: We plot the log MSE for off-policy evaluation in the image-based Atari domain, using a distinct behavior policy, trained by a separate algorithm, from the target policy. This experiment tests the ability to generalize to a more off-policy setting. The shaded area captures one standard deviation across 3 trials. Markers are not placed at every point for visual clarity.
Again, we observe similar trends in performance. Notably, in the Asterix game, the performance of Deep SR surpasses the behavior policy, suggesting off-policy evaluation can outperform the naïve estimator in settings where the policy is sufficiently "off-policy" and distinct.

C. SR-DICE Practical Details
In this section, we cover some basic implementation-level details of SR-DICE. Note that code is provided for additional clarity.
SR-DICE uses two parametric networks, an encoder-decoder network to learn the encoding φ and a deep successor representation network ψ π . Additionally, SR-DICE uses the weights of a linear function w. SR-DICE begins by pre-training the encoder-decoder network and the deep successor representation before applying updates to w.
Encoder-Decoder. This encoder-decoder network encodes (s, a) to the feature vector φ(s, a), which is then decoded by several decoder heads. For the Atari domain, we choose to condition the feature vector only on states φ(s), as the reward is generally independent of the action selection. This change applies to both SR-DICE and Deep SR. Most design decisions

Algorithm 2 SR-DICE
Input: dataset D, target policy π, number of iterations T 1 , T 2 , T 3 , mini-batch size N , target update rate.
# Train the encoder-decoder. for t = 1 to T 1 do Sample mini-batch of N transitions (s, a, r, s ) from D. min φ,D s ,Da,Dr λ s (D s (φ(s, a)) − s ) 2 +λ a (D a (φ(s, a)) − a) 2 + λ r (D r (φ(s, a)) − r) 2 . end for # Train the successor representation. for t = 1 to T 2 do Sample mini-batch of N transitions (s, a, r, s ) from D.
For continuous control, given a mini-batch transition (s, a, r, s ), the encoder-decoder network is trained to map the state-action pair (s, a) to the next state s , the action a and reward r. The resulting loss function is as follows: We use λ s = 1, λ a = 1 and λ r = 0.1.
For the Atari games, given a mini-batch transition (s, a, r, s ), the encoder-decoder network is trained to map the state s to the next state s and reward r, while penalizing the size of φ(s). The resulting loss function is as follows: We use λ s = 1, λ r = 0.1 and λ φ = 0.1.
Deep Successor Representation. The deep successor representation ψ π is trained to estimate the accumulation of φ. The training procedure resembles standard deep reinforcement learning algorithms. Given a mini-batch of transitions (s, a, r, s ) the network is trained to minimize the following loss: where ψ is the target network. A target network is a frozen network used to provide stability (Mnih et al., 2015;Kulkarni et al., 2016) in the learning target. The target network is updated to the current network ψ ← ψ π after a fixed number of time steps, or updated with slowly at each time step ψ ← τ ψ π + (1 − τ )ψ π (Lillicrap et al., 2015).
Evaluation. The marginalized importance sampling methods are measured by the average weighted reward from transitions sampled from a replay buffer 1 N (s,a,r) w(s, a)r(s, a), with N = 10k, while the deep RL methods use (1−γ) M s0 Q(s 0 , π(a 0 )), where M is the number of episodes. Each OPE method is trained on data collected by some behavioral policy π b . We estimate the "true" normalized average discounted reward of the target and behavior policies from 100 roll-outs in the environment.

E.1. Continuous Action Environments
Our agents are evaluated via tasks interfaced through OpenAI gym (version 0.17.2) (Brockman et al., 2016), which mainly rely on the MuJoCo simulator (mujoco-py version 1.50.1.68) (Todorov et al., 2012). We provide a description of each environment in Table 1. Experiments. Our experiments are framed as off-policy evaluation tasks in which agents aim to evaluate R(π) = E (s,a)∼d π ,r [r(s, a)] for some target policy π. In each of our experiments, π corresponds to a noisy version of a policy trained by a TD3 agent (Fujimoto et al., 2018), a commonly used deep reinforcement learning algorithm. Denote π d , the deterministic policy trained by TD3 using the author's GitHub https://github.com/sfujim/TD3. The target policy is defined as: π + N (0, σ 2 ), where σ = 0.1. The off-policy evaluation algorithms are trained on a dataset generated by a single behavior policy π b . The experiments are done with two settings easy and hard which vary the behavior policy and the size of the dataset. All other settings are kept fixed. For the easy setting the behavior policy is defined as: and 500k time steps are collected (approximately 500 trajectories for most tasks). The easy setting is roughly based on the experimental setting from Zhang et al. (2020a). For the hard setting the behavior policy adds an increased noise and selects random actions with p = 0.2: and only 50k time steps are collected (approximately 50 trajectories for most tasks). For Pendulum-v0 and Humanoid-v3, the range of actions is [−2, 2] and [−0.4, 0.4] respectively, rather than [−1, 1], so we scale the size of the noise added to actions accordingly. We set the discount factor to γ = 0.99. All continuous action experiments are over 10 seeds.
Pre-training. Both SR-DICE and Deep SR rely on pre-training the encoder-decoder and deep successor representation ψ. These networks were trained for 30k and 100k time steps respectively. As noted in Section B.4, even when including this pre-training step, both algorithm have a lower running time than DualDICE and GradientDICE.
Architecture. For fair comparison, we use the same architecture for all algorithms except for DualDICE. This a fully connected neural network with 2 hidden layers of 256 and ReLU activation functions. This architecture was based on the network defined in the TD3 GitHub and was not tuned. For DualDICE, we found tanh activation functions improved stability over ReLU.
For SR-DICE and SR-Direct we use a separate architecture for the encoder-decoder network. The encoder is a network with a single hidden layer of 256, making each φ(s, a) a feature vector of 256. There are three decoders for reward, action, and next state, respectively. For the action decoder and next state decoder we use a network with one hidden layer of 256.
The reward decoder is a linear function of the encoding, without biases. All hidden layers are followed by ReLU activation functions.
Network hyper-parameters. All networks are trained with the Adam optimizer (Kingma & Ba, 2014). We use a learning rate of 3e−4, again based on TD3 for all networks except for GradientDICE, which we found required careful tuning to achieve a reasonable performance. For GradientDICE we found a learning rate of 1e−5 for f and w, and 1e−2 for u achieved the highest performance. For DualDICE we chose the best performing learning rate out of {1e − 2, 1e − 3, 3e − 4, 5e − 5, 1e − 5}. SR-DICE, Deep SR, and Deep TD were not tuned and use default hyper-parameters from deep RL algorithms. For training ψ π and Q π for the deep reinforcement learning aspects of SR-DICE, Deep SR, and Deep TD we use a mini-batch size of 256 and update the target networks using τ = 0.005, again based on TD3. For all MIS methods, we use a mini-batch size of 2048 as described by (Nachum et al., 2019a). We found SR-DICE and DualDICE succeeded with lower mini-batch sizes but did not test this in detail. All hyper-parameters are described in Table 2.  Mini-batch size  2048  2048  2048 2048 ψ π , Q π Target update rate 0.005 --0.005 0.005 Visualizations. We graph the log MSE between the estimate of R(π) and the true R(π), where the log MSE is computed as log 0.5(X − R(π)) 2 . We smooth the learning curves over a uniform window of 10. Agents were evaluated every 1k time steps and performance is measured over 250k time steps total. Markers are displayed every 25k time steps with offset for visual clarity.
Randomized reward experiments. The randomized reward experiment uses the MIS ratios collected from the hard setting from the continuous action environments. 1000 reward functions were generated by a randomly initialized neural network. The neural network architecture is two hidden layers of 256 with ReLU activation functions after each hidden layer and a sigmoid activation function after the final layer. Weights were sampled from the normal distribution and biases were set to 0. The ground truth value was estimated from 100 on-policy trajectories. If the ground truth value was less than 0.1 or greater than 0.9, (where the range of possible values is [0, 1]), the reward function was considered redundant and removed. To reduce variance across reward functions, we normalize both the estimated R(π) and ground truth R(π) by dividing by the average reward for all state-action pairs in the dataset. Pre-processing. We use standard pre-processing steps based on Machado et al. (2018b) and Castro et al. (2018). We base our description on (Fujimoto et al., 2019a), which our code is closely based on. We define the following:

E.2. Atari
• Frame: output from the Arcade Learning Environment.
• State: conventional notion of a state in a MDP.
• Input: input to the network.
The notion of time steps is applied to states, rather than frames, and functionally, the concept of frames can be abstracted away once pre-processing has been applied to the environment.
The agent receives a state every 4th frame and selects one action, which is repeated for the following 4 frames. If the environment terminates within these 4 frames, the state received will be the last 2 frames before termination. For the first 3 time steps of an episode, the input, which considers the previous 4 states, sets the non-existent states to all 0s. An episode terminates after the game itself terminates, corresponding to multiple lives lost (which itself is game-dependent), or after 27k time steps (108k frames or 30 minutes in real time). Rewards are clipped to be within a range of [−1, 1].
Sticky actions are applied to the environment (Machado et al., 2018b), where the action a t taken at time step t, is set to the previously taken action a t−1 with p = 0.25, regardless of the action selected by the agent. Note this replacement is abstracted away from the agent and dataset. In other words, if the agent selects action a at state s, the transition stored will contain (s, a), regardless if a is replaced by the previously taken action.
Experiments. For the main experiments we use a behavior and target policy derived from a Double DQN agent (Van Hasselt et al., 2016), a commonly used deep reinforcement learning algorithm. The behavior policy is an -greedy policy with = 0.1 and the target policy is the greedy policy (i.e. = 0). In Section B.5 we perform two additional experiments with a different behavior policy. Otherwise, all hyper-parameters are fixed across experiments. For each, the dataset contains 1 million transitions and uses a discount factor of γ = 0.99. Each experiment is evaluated over 3 seeds.
Pre-training. Both SR-DICE and Deep SR rely on pre-training the encoder-decoder and deep successor representation ψ. Similar to the continuous action tasks, these networks were trained for 30k and 100k time steps respectively.
Architecture. We use the same architecture as most value-based deep reinforcement learning algorithms for Atari, e.g. (Mnih et al., 2015;Van Hasselt et al., 2016;Schaul et al., 2016). This architecture is used for all networks, other than the encoder-decoder network, for fair comparison and was not tuned in any way.
The network has a 3-layer convolutional neural network (CNN) followed by a fully connected network with a single hidden layer. As mentioned in pre-processing, the input to the network is a tensor with shape (4, 84, 84). The first layer of the CNN has a kernel depth of 32 of size 8 × 8 and a stride of 4. The second layer has a kernel depth of 32 of size 4 × 4 and a stride of 2. The third layer has a kernel depth of 64 of size 3 × 3 and a stride of 1. The output of the CNN is flattened to a vector of 3136 before being passed to the fully connected network. The fully connected network has a single hidden layer of 512. Each layer, other than the output layer, is followed by a ReLU activation function. The final layer of the network outputs |A| values where |A| is the number of actions.
The encoder-decoder used by SR-DICE and SR-Direct has a slightly different architecture. The encoder is identical to the aforementioned architecture, except the final layer outputs the feature vector φ(s) with 256 dimensions and is followed by a ReLU activation function. The next state decoder uses a single fully connected layer which transforms the vector of 256 to 3136 and then is passed through three transposed convolutional layers each mirroring the CNN. Hence, the first layer has a kernel depth of 64, kernel size of 3 × 3 and a stride of 1. The second layer has a kernel depth of 32, kernel size of 4 × 4 and a stride of 2. The final layer has a kernel depth of 32, kernel size of 8 × 8 and a stride of 4. This maps to a (1, 84, 84) tensor. All layers other than the final layer are followed by ReLU activation functions. Although the input uses a history of the four previous states, as mentioned in the pre-processing section, we only reconstruct the succeeding state without history. We do this because there is overlap in the history of the current input and the input corresponding to the next time step. The reward decoder is a linear function without biases.
Network hyper-parameters. Our hyper-parameter choices are based on standard hyper-parameters based largely on (Castro et al., 2018). All networks are trained with the Adam optimizer (Kingma & Ba, 2014). We use a learning rate of 6.25e−5.
Although not traditionally though of has a hyper-parameter, in accordance to prior work, we modify used by Adam to be 1.5e−4. For w we use a learning rate of 3e−4 with the default setting of = 1e−8. For u we use 1e−3. We use a mini-batch size of 32 for all networks. SR-DICE, Deep SR, and Deep TD update the target network every 8k time steps. All hyper-parameters are described in Table 3.
Visualizations. We use identical visualizations to the continuous action environments. Graphs display the log MSE between the estimate of R(π) and the true R(π) of the target policy, where the log MSE is computed as log 0.5(X − R(π)) 2 . We smooth the learning curves over a uniform window of 10. Agents were evaluated every 1k time steps and performance is measured over 250k time steps total. Markers are displayed every 25k time steps with offset for visual clarity.