{'fraction_non_alphanumeric': 0.033374157991426824, 'fraction_numerical': 0.010293645933572449, 'mean_word_length': 4.633429158110883, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 10, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Models of human preference for learning reward functions

The utility of reinforcement learning is limited by the alignment of reward functions with the interests of human stakeholders. One promising method for alignment is to learn the reward function from human-generated preferences between pairs of trajectory segments. These human preferences are typically assumed to be informed solely by partial return, the sum of rewards along each segment. We find this assumption to be flawed and propose modeling preferences instead as arising from a different statistic: each segment's regret, a measure of a segment's deviation from optimal decision-making. Given infinitely many preferences generated according to regret, we prove that we can identify a reward function equivalent to the reward function that generated those preferences. We also prove that the previous partial return model lacks this identifiability property without preference noise that reveals rewards' relative proportions, and we empirically show that our proposed regret preference model outperforms it with finite training data in otherwise the same setting. Additionally, our proposed regret preference model better predicts real human preferences and also learns reward functions from these preferences that lead to policies that are better human-aligned. Overall, this work establishes that the choice of preference model is impactful, and our proposed regret preference model provides an improvement upon a core assumption of recent research.


Introduction
Improvements in reinforcement learning (RL) have led to notable recent achievements [1][2][3][4][5][6], increasing its applicability to real-world problems. Yet, like all optimization algorithms, even perfect RL optimization is limited by the objective it optimizes. For RL, this objective is created in large part by the reward function. Poor alignment between reward functions and the interests of human stakeholders limits the utility of RL and may even pose catastrophic risks [7,8].
Influential recent research has focused on reward learning from preferences over pairs of fixed-length trajectory segments. Nearly all of this recent work assumes that human preferences arise probabilistically from only the sum of rewards over a segment, i.e., the segment's partial return [9][10][11][12][13][14][15][16]. That is, these works assume that people tend to prefer trajectory segments that yield greater rewards during the segment. However, this preference model ignores seemingly important information about the segment's desirability, including the state values of the segment's start and end states. Separately, this partial return preference model can prefer suboptimal actions with lucky outcomes, like buying a lottery ticket. This paper proposes an alternative preference model based on the regret of each segment, which is equivalent to the negated sum of an optimal policy's advantage of each transition in the segment (Section 2.2). Figure 1 shows an intuitive example of when these two models disagree. Other classes of domains that the models will differ on are those with constant reward until the end, including competitive games like chess, go, and soccer as well as tasks for which the objective is to minimize time until reaching a goal. Figure 1: Two segments of a car moving at high speed near a brick wall. Assume the right segment is optimal and the left segment is suboptimal (as defined in Sec. 2.1). The left segment has a higher sum of reward, so the partial return preference model tends to prefer it. The regret preference model instead tends to prefer the right segment because optimal segments have minimal regret. If we also assume deterministic transitions, then the regret model includes the difference in values between the start state and the end state (Eq. 3), and the right segment would tend to be preferred because it greatly improves its state values from start to end, whereas the left segment's state values greatly worsen. We suspect our human readers will also tend to prefer the right segment.
For these two preference models, we first focus theoretically on a normative analysis (Section 3)-i.e., what preference model would we want humans to use if we could choose-proving that reward learning on infinite, exhaustive preferences with our proposed regret preference model identifies a reward function with the same set of optimal policies as the reward function with which the preferences are generated. We also prove that the partial return preference model is not guaranteed to identify such a reward function without preference noise. We follow up with a descriptive analysis of how well each of these proposed models align with actual human preferences by collecting a humanlabeled dataset of preferences in a rich grid world domain (Section 4) and showing that the regret preference model better predicts these human preferences (Section 5). Finally, we find that the policies ultimately created through the regret preference model tend to outperform those from the partial return model learning-both when assessed with collected human preferences or when assessed with synthetic preferences (Section 6).

Preference models for learning reward functions
We assume that the task environment is a Markov decision process (MDP) specified by the tuple (S, A, T , γ, D 0 , r). S and A are the sets of possible states and actions, respectively. T is a transition function, T : S ×A× → S. γ is the discount factor and D 0 is the distribution of start states. Unless otherwise stated, we assume undiscounted tasks (i.e., γ = 1) that have terminal states, after which only 0 reward can be received. r is a reward function, r : S ×A×S → R, where the reward r t at time t is a function of s t , a t , and s t+1 . An MDP\r is an MDP without a reward function.
Throughout this paper, r refers to the ground-truth reward function for some MDP;r refers to a learned approximation of r; andr refers to any reward function (including r orr). A policy (π : S ×A → [0,1]) specifies the probability of an action given a state. Q * r and V * r refer respectively to the state-action value function and state value function for an optimal policy, π * , underr. The optimal advantage function is defined as A * r (s,a) Q * r (s,a)−V * r (s). Throughout this paper, the ground-truth reward function r is used to algorithmically generate preferences when they are not human-generated, is hidden during reward learning, and is used to evaluate the performance of optimal policies under a learnedr.

Reward learning from pairwise preferences
A reward function can be learned by minimizing the cross-entropy loss-i.e., maximizing the likelihood-of observed human preferences, a common approach in recent literature [9-11, 14, 16].
Loss function To learn a reward function from a preference dataset, D , a common assumption is that these preferences were generated by a preference model P that arises from an unobservable ground-truth reward function r.We approximate r by minimizing cross-entropy loss to learnr: loss(r,D ) = − (σ1,σ2,µ)∈D µ 1 logP (σ 1 σ 2 |r)+µ 2 logP (σ 1 ≺ σ 2 |r) This loss is under-specified until P (σ 1 σ 2 |r) is defined, which is the focus of this paper. We show that the common model of preference probabilities is flawed and introduce an improved preference model.
Preference models A preference model determines the probability of one trajectory segment being preferred over another, P (σ 1 σ 2 |r). Preference models could be applied to model preferences provided by humans or other systems. Preference models can also directly generate preferences, and in such cases we refer to them as preference generators.

Choice of preference model: partial return and regret
Partial return Recent work assumes human preferences are generated by a Boltzmann distribution over the two segments' partial returns [9][10][11][12][13][14][15][16], expressed here as a logistic function 4 : Regret We introduce an alternative preference model based on the regret of each transition in a segment. We first focus on segments with deterministic transitions. For a transition (s t ,a t ,s t+1 ) in a deterministic segment, regret d (σ t |r) V * r (s σ,t )−[r t +V * r (s σ,t+1 )]. For a full deterministic segment, with the right-hand expression arising from cancelling out intermediate state values. Therefore, deterministic regret measures how much the segment reduces expected return from V * r (s σ,0 ). An optimal segment, σ * , always has 0 regret, and a suboptimal segment, σ ¬ * , will always have positive regret, a intuitively appealing property that also plays a role in the identifiability proof of Theorem 3.1.
Stochastic transitions, however, can result in regret d (σ * |r) > regret d (σ ¬ * |r), losing the property above. To retain it, we note that the effect on expected return of transition stochasticity from a transition (s t ,a t ,s t+1 ) is [r t +V * r (s t+1 )]−Q * r (s t ,a t ) and add this expression once per transition to get regret(σ), removing the subscript d that refers to determinism. The regret for a single transition becomes regret(σ t |r) Regret for a full segment is The regret preference model is the Boltzmann distribution over negated regret: P regret (σ 1 σ 2 |r) logistic regret(σ 2 |r)−regret(σ 1 |r) .
Lastly, we note that if two segments have deterministic transitions, end in terminal states, and have the same starting state, this regret model reduces to the partial return model: P regret (·|r) = P Σr (·|r).
Regret as a model for human preference P regret makes at least three assumptions worth noting. First, it keeps the assumption that human preferences follow a Boltzmann distribution over some statistic, which is a common model of choice behavior in economics and psychology, where it is called the Luce-Shepard choice rule [17,18]. Second, P regret implicitly assumes humans can identify optimal and suboptimal segments when they see them, which will less true in domains where the human has less expertise. Lastly, P regret assumes that in stochastic settings where the best outcome may only result from suboptimal decisions (e.g., buying a lottery ticket), humans instead prefer optimal decisions. We suspect humans are capable of expressing either type of preference-based on decision quality or desirability of outcomes-and can be influenced by training or the preference elicitation interface.
In practice we determine that the regret model produces improvements over the partial-return model (Section 6), and its assumptions represent an opportunity for follow-up research.
Alternative methods for learning reward functions Other methods for learning reward functions include inverse reinforcement learning from demonstrations [19,20] and inverse reward design from trial-and-error reward design in multiple instances of a task domain [21].

Theoretical comparisons
In this section, we consider how different ways of generating preferences affect reward inference, setting aside whether humans can be influenced to give preferences in accordance with a specific preference method. In economic terms, this analysis-and all of our analyses with synthetic preferences-could be considered a normative analysis. In artificial intelligence, this analysis might be cast as a step towards defining criteria for a rational preference model.
. For a preference model P , assume an infinite dataset D of n-length pairs of segments is constructed by repeatedly choosing (σ 1 ,σ 2 ) and sampling a label µ ∼ P (σ 1 σ 2 |r), using P as a preference generator. Further assume that in this dataset, all possible n-length segment pairs appear infinitely many times. For some MDP\r M , let Mr be M with the reward functionr. Let Π * r be the set of optimal policies in Mr. Let reward-equivalence class R be the set of all reward functions such that if r 1 ,r 2 ∈ R then Π * r1 = Π * r2 . Preference model P is identifiable if, for any choice of n and M r , anyr = argminr ,D [loss(r)]-for the cross-entropy loss (Eqn. 1), with P as the preference model-is in the same reward equivalence class as r. I.e., Π * r = Π * r . Theorem 3.1 (P regret is identifiable). Let P regret be any function such that if regret(σ 1 |r) < regret(σ 2 |r), P regret (σ 1 σ 2 |r) > 0.5, and if regret(σ 1 |r) = regret(σ 2 |r), P regret (σ 1 σ 2 |r) = 0.5. P regret is identifiable.
This class of regret preference models includes but is not limited to the Boltzmann distribution of Eqn. 5.
Proof sketch Make all assumptions in Definition 3.1. Sincer minimizes cross-entropy loss, P regret (·|r) = P regret (·|r) for all possible segment pairs. Also, by Equation 4 regret(σ|r) = 0 if and only if σ is optimal with respect tor. And regret(σ|r) > 0 if and only if σ is suboptimal withr.
Appendix C contains a detailed proof of Theorem 3.1 and two proofs by example for Theorem 3.2, each focusing on a different weakness of P Σr .The first proof by example reveals issues when learning reward functions with stochastic transitions with either P Σr or deterministic P regret d . These issues directly correspond to the need for preferences over distributions over outcomes (i.e., lotteries) to construct a cardinal utility function (see Russell and Norvig [22,Ch. 16]). Note that the noiseless version of P Σr in Theorem 3.2 is achieved in the limit as reward values are scaled higher; equivalently, one could include a Boltzmann temperature parameter in Equation 2 and scale it towards 0. Intuitively, Theorem 3.2 says that P Σr is not identifiable without the distribution over preferences providing information about the proportions of rewards with respect to each other. In contrast, to be identifiable, the regret preference model does not require this preference error (though it can presumably benefit from it in certain contexts).

Creating a human-labeled preference dataset
To empirically investigate the consequences of each preference model when learning reward from human preferences, we created a preference dataset labeled by human subjects via Amazon Mechanical Turk. Appendix D adds detail to the content below.

The general delivery domain
The delivery domain consists of a grid of cells, each of a specific road surface type. The delivery agent's state is its location. The agent's action space is moving one cell in one of the four cardinal directions. The episode can terminate either at the destination for +50 reward or in failure at a sheep for −50 reward. The reward for a non-terminal transition is the sum of any reward components. Cells with a white road surface have a −1 reward component, and cells with brick surface have a −2 component. Additionally, each cell may contain a coin (+1) or a roadblock (−1). Coins do not disappear and at best cancel out the road surface cost. Actions that would move the agent into a house or beyond the grid's perimeter result in no motion and receive reward that includes the current cell's surface reward component but not any coin or roadblock components. In this work, the start state distribution, D 0 , is always uniformly random over non-terminal states. This domain was designed to permit subjects to easily identify bad behavior yet also to be difficult for them to determine optimal behavior from most states, which is representative of many common tasks.  We chose one instantiation of the delivery domain for gathering our dataset of human preferences. This specific MDP has a 10×10 grid. From every state, the highest return possible involves reaching the goal, rather than hitting a sheep or perpetually avoiding termination. Figure 2 shows this task.

The user interface and survey
This subsection describes the three main stages of the experimental session. A video showing the full experimental protocol can be seen at bit.ly/3FZq5ZP. Teaching subjects about the task Subjects first view instructions describing the general domain. To avoid the jargon of "return" and "reward," these terms are mapped to equivalent values in US dollars, and the instructions describe the goal of the task as maximizing the delivery vehicle's financial outcome, where the reward components are specific financial impacts. This information is shared amongst interspersed interactive episodes, in which the subject controls the agent in domain maps that are each designed to teach one or two concepts. Our intention during this stage is to inform the later preferences of the subject by teaching them about the domain's dynamics and its reward function, as well as to develop the subject's sense of how desirable various behaviors are. At the end of this stage, the subject controls the agent for two episodes in the specific delivery task shown in Figure 2.
Preference elicitation After each subject is trained to understand the task, they indicate their preferences between 40-50 randomly-ordered pairs of segments, using the interface shown in Figure 3. The users select a preference, no preference ("same"), or "can't tell". In this work, we exclude responses labeled "can't tell", though one might alternatively try to extract information from these responses.
Users' task comprehension Subjects then answered questions testing their understanding of the task, and we removed their data if they scored poorly. We also removed a subject's data if they preferred colliding the vehicle into a sheep over not doing so, which we interpreted as poor task understanding or inattentiveness. This filtered dataset contains 1812 preferences from 50 subjects. We collected human preferences in two stages, each with different methods for selecting which segment pairs to present for labeling. The second stage's sole purpose was to improve the reward-learning performance of P Σr . Before this second stage, P Σr compared even worse to P regret than in the results described in Section 6. These methods and their justification are described in Appendix D.3. This section considers how well different preference models explain our dataset of human preferences.

Correlations between preferences and segment statistics
We hypothesize that the values of segments' start and end states-which are included in P regret but not in P Σ -affect human preferences, independent of partial return. To simplify analysis, we combine the two parts of regret d (σ|r) that are additional to Σ σr and introduce the following shorthand: . Therefore, on the diagonal line in Figure 4, regret d (σ 2 |r) = regret d (σ 1 |r), making the P regret d preference model indifferent.

Preference model
Loss P (·) = 0.5 (uninformed) 0.69 PΣ r (partial return) 0.62 Pregret 0.57 The dataset of preferences is visualized in Figure 4. This plot shows how ∆ σ V r has influence independent of partial return by focusing only on points at a chosen y-axis value; if the colors along the corresponding horizontal line reddens as the x-axis value increases, then ∆ σ V r appears to have independent influence. To statistically test for independent influence of ∆ σ V r on preferences, we consider subsets of data where Σ σ1 r−Σ σ2 r is constant. For Σ σ1 r −Σ σ2 r = −1 and Σ σ1 r −Σ σ2 r = −2, the only values with more than 30 samples that also include informative samples with both negative and positive values of regret(σ 1 |r)−regret(σ 2 |r), the Spearman's rank correlations between ∆ σ V r and the preferences are significant (r >= 0.3, p < 0.0001). This result indicates that ∆ σ V r influences human preferences independent of partial return, validating our hypothesis that humans form preferences based on information about segments' start states and end states, not only partial returns.

Likelihood of human preferences under different preference models
To examine how well each preference model predicts human preferences, we calculate the crossentropy loss for each model (Eqn. 1)-i.e., the negative log likelihood-of the preferences in our dataset. Scaling reward by a constant factor does not affect the set of optimal policies. Therefore, throughout this work we ensure that our analyses of preference models are insensitive to reward scaling. To do so for this specific analysis, we conduct 10-fold cross validation to learn a reward scaling factor for each of P regret and P Σr . Table 1 shows that the loss of P regret is lower than that of P Σr , indicating that it is more reflective of how people actually express preferences.

Results from learning reward functions
Analysis of a preference model's predictions of human preferences is informative, but such predictions are a means to the ends of learning human-aligned reward functions and policies. We now examine each preference model's performance on these ends. In all cases, we learn a reward functionr according to Eqn. 1 and apply value iteration [23] to find the approximately optimal Q * r function. For this Q * r , we then evaluate the mean return of the maximum-entropy optimal policy-which chooses uniformly randomly among all optimal actions-with respect to the ground-truth reward function r, over D 0 . To compare performance across different MDPs, the mean return of a policy π, V π r , is normalized to where V * r is the optimal expected return and V U r is the expected return of the uniformly random policy (both given D 0 ). Normalized mean return above 0 is better than V U r . Optimal policies have a normalized mean return of 1, and we consider above 0.9 to be near optimal.
6.1 An algorithm to learn reward functions with regret(σ σ |r) Algorithm 1 is a general algorithm for learning a linear reward function according to P regret . Adding to our assumptions listed in Section 2.2, this algorithm introduces two more approximations. First, to avoid expensive evaluation of V * r (s σ,t ) and Q * r (s σ,t ,a σ,t ) to compute P regret (·|r) during reward learning, successor features for a set of policies are used to approximate the optimal state values and state-action values for any reward function. Second, softmax weighting is used, as explained below.
Approximating P regret with successor features Following the notation of Barreto et al. [24], assume the ground-truth reward is linear with respect to a feature vector extracted by φ : S ×A×S → R d and a weight vector w r ∈ R d : r(s,a,s ) = φ(s,a,s ) w r .During learning, wr similarly expressesr aŝ r(s,a,s ) = φ(s,a,s ) wr.
Given a policy π, the successor features for (s,a) are the expectation of discounted reward features from that state-action pair when following π: wr. Additionally, state-based successor features can be calculated from the ψ π Q above as ψ π V (s) = a∈A π(a|s)ψ π Q (s,a), making V π r (s) = ψ π V (s) wr. Given a set Ψ Q of state-action successor feature functions and a set Ψ V of state successor feature functions for various policies and given a reward function via wr, ,so we use these two maximizations as approximations of Q * r (s,a) and V * r (s), respectively. In practice, to enable gradient-based optimization with current tools, the maximization in this expression is replaced with the softmax-weighted average, making the loss function linear. Focusing first on the approximation of V * r (s), for each if r SF then π SF ← estimate of optimal maximum-entropy policy for r SF

5:
estimate ψ π SF Q and ψ π SF V (if not estimated already during step 4) 6: add ψ π SF Q to Ψ Q 7: add ψ π SF V to Ψ V 8: end for 9: repeat 10: optimize wr by loss of Eqn. 1, calculatingP regret (σ 1 σ 2 |r) via Eqn. 6, using Ψ Q and Ψ V 11: until stopping criteria are met 12: return wr and 5, the corresponding approximationP regret of the regret preference model is: The algorithm In Algorithm 1, lines 9-12 describe the supervised-learning optimization using the approximationP regret , and the prior lines create Ψ Q and Ψ V . Specifically, given a set of reward functions, a corresponding set of policies is created (line 4), where each policy is an estimate of the maximum entropy policy for a reward function. Standard policy improvement methods can be used to create each such policy. Alternatively, some or all of the set of policies can be given as input directly, not derived from input reward functions. For each such policy π SF , successor feature functions Ψ π SF Q and Ψ π SF V are estimated (line 5), which by default would be performed by a minor extension of a standard policy evaluation algorithm as detailed by Barreto et al. [24]. Note that the reward function that is ultimately learned is not restricted to be in the input set of reward functions, which is used only to create an approximation of regret.
The details of our instantiation of Algorithm 1 for the delivery domain can be found in Appendix F.1, along with guidance for extending it to reward functions that might be non-linear.

Results from synthetic preferences
Before considering human preferences, we first ask how each preference model performs when it is correct. In other words, we investigate empirically how well the preference model could perform if humans perfectly adhered to it. Recall that the ground-truth reward function, r, is used to create these preferences but is inaccessible to the reward-learning algorithms. For these evaluations, either a stochastic or noiseless preference model acts a preference generator to create a preference dataset, and then the stochastic version of the same model is used for reward learning.For the noiseless case, the deterministic preference generator compares a segment pair's Σ σ r values for P Σr or their regret(σ|r) values for P regret . Note that through reward scaling the preference generators approach determinism in the limit, so this noiseless analysis examines minimal-entropy versions of the two preference-generating models. (The opposite extreme, uniformly random preferences, would remove all information from preferences and therefore is not examined.) In the stochastic case, for each preference model, each segment pair is labeled by sampling from that preference generator's output distribution (Eqs 2 or 5), using the unscaled ground-truth reward function.
We created 100 deterministic MDPs that instantiate variants of our delivery domain (see Section 4.1). To create each MDP, we sampled from sets of possible widths, heights, and reward component values, and the resultant grid cells were randomly populated with a destination, objects, and road surface types (see Appendix F.2 for details). Each segment in the preference datasets for each MDP was generated by choosing a start state and three actions, all uniformly randomly. For a set number of preferences, each method had the same set of segment pairs in its preference dataset. Figure 5 shows the percentage of MDPs in which each preference model results in near-optimal performance. The regret preference model outperforms the partial return model at every dataset size, both with and without noise. By a Wilcoxon paired signed-rank test on normalized mean returns, p < 0.05 for 86% of these comparisons and p < 0.01 for 57% of them, as reported in Appendix F.2.
Further analyses can be found in Appendix F.2, including with stochastic transitions, with different segment lengths, and while artificially lowering the discount factor (as is common in deep RL and recent work on deep reward learning from preferences). We randomly assign human preferences from our gathered dataset to different numbers of same-sized partitions, resulting in different training set sizes, and test each preference model on each partition. Figure 6 shows the results. With smaller training sets (20-100 partitions), the regret preference model results in nearoptimal performance more often. With larger training sets (1-10 partitions), both preference models always reach near-optimal return, but the mean return from the regret preference model is higher for all of these partitions except for 3 partitions in the 10-partition test. Applying a Wilcoxon paired signed-rank test on normalized mean return to each group with 5 or more partitions, p < 0.05 for all numbers of partitions except 100 and p < 0.01 for 20 and 50 partitions.

Conclusion
Over numerous evaluations with human preferences, our proposed regret preference model (P regret ) shows improvements summarized below over the previous partial return preference model (P Σr ).When each preference model generates the preferences for its own infinite and exhaustive training set, we prove that P regret identifies the set of optimal policies, whereas P Σr is not guaranteed to do so without preference noise that reveals the proportions of rewards with respect to each other. With finite training data of synthetic preferences, P regret also empirically results in learned policies that tend to outperform those resulting from P Σr . This superior performance of P regret is also seen with human preferences. In summary, our analyses suggest that regret preference models are more effective both descriptively with respect to human preferences and also normatively, as the model we want humans to follow if we had the choice. Independent of P regret , this paper also reveals that segments' changes in state values provide information about human preferences that is not fully provided by partial return. More generally, we show that the choice of preference model impacts the performance of learned reward functions.
This study motivates several new directions for research. Future work could address any of the limitations detailed in Appendix A.1. Specifically, future work could further test the general superiority of P regret or apply it to deep learning settings. Additionally, prescriptive methods could be developed via the user interface or elsewhere to nudge humans to conform more to P regret or to other normatively appealing preference models. Lastly, subsequent efforts could seek preference models that are even more effective with preferences from actual humans, now that this work has provided conclusive evidence that the choice of preference model is impactful.
After Appendix A, the appendix is organized according to the major sections and subsections of the main content. We consider Appendix C to be its most important addition to the main content.

A.1 Limitations
Some limitations of the regret preference model are discussed in the paragraph "Regret as a model for human preference" in Section 2.2, including assumptions that a person giving preferences can distinguish between optimal and suboptimal segments, that they follow a Boltzmann distribution (i.e., a Luce Shepard choice rule), and that they base their preferences on the desirability of decisions even when transition stochasticity results in segment pairs for which the worse decision has a better outcome.
Our proposed algorithm (Section 6.1) has a few additional limitations. Generating candidate successor features for the approximationsQ * r andṼ * r may be difficult in complex domains. Specifically, challenges include choosing the set of policies or reward functions for which to compute successor features (line 3 of Algorithm 1) and creating a reward feature vector φ for non-linear reward functions (discussed in Appendix F.1). Additionally, although learning with P regret is more sample efficient in our experiments, it is computationally slower than learning with P Σr because of the additional need to compute successor features and the use of the softmax function to approximate Q * r and V * r . Nonetheless, we may accept the tradeoff of an increase in computational time that reduces the number of human samples needed or that improves the reward function's alignment with human stakeholders' interests. Lastly, the loss during optimization with P regret was unstable, which we addressed by taking the minimum loss over all epochs during training. Therefore, for more complex reward feature vectors (φ) than our 6-element vector for the delivery task, extra care might be needed to avoid overfittingr, for example by withholding some preference data to serve as a test set.
We also generally assume that the RL algorithm and reward learning algorithm use the same discount factor as in the MDP\r specification. One weakness of contemporary deep RL is that RL algorithms require artificially lower discount factors than the true discount factor of the task. The interaction of this discounting with preference models is considered in Appendix F.2. Our expectation though is that this weakness of deep RL algorithms is likely a temporary one, and so we focused our analysis on simple tasks in which we do not need to artificially lower the RL algorithm's discount factor. However, further investigation of the interaction between preference models and discount factors would aid near-term application of P regret to deep RL domains.
This work also does not consider which segment pairs should be presented for labeling with preferences used for reward learning. However, other research has addressed this problem through active learning [14][9] [25], and it may be possible to simply swap our Algorithm 1 into these active learning methods, combining the improved sample efficiency of P regret with that of these active learning methods.
Regarding the human side of the problem of reward learning from preferences, further research could provide several improvements. First, we are confident that humans can be influenced by their training and by the preference elicitation interface, which is a particularly rich direction for follow-up study. We also do not consider how to handle learning reward functions from multiple human stakeholders who have different preferences, a topic we revisit in Appendix A.2. Lastly, we expect humans to deviate from any simple model, including P regret , and a fine-grained characterization of how humans generate preferences could produce preference models that further improve the alignment of the reward functions that are ultimately learned from human preferences.

A.2 Ethical statement
This work is meant to address ethical issues that arise when autonomous systems are deployed without properly aligning their objectives with those of human stakeholders. It is merely a step in that direction, and overly trusting in our methods-even though they improve on previous methods for alignmentcould result in harm caused by poorly aligned autonomous systems.
When considering the objectives for such systems, a critical ethical question is which human stakeholders' interests the objectives should be aligned with and how multiple stakeholders' interests should be combined into a single objective for an autonomous system. We do not address these important questions, instead making the convenient-but-flawed assumption that many different humans' prefer-ences can simply be combined. In particular, care should be taken that vulnerable and marginalized communities are adequately represented in any technique or deployment to learn a reward function from human preferences in high-impact settings. The stakes are high: for example, a reward function that is only aligned with a corporation's financial interests could lead to exploitation of such communities or more broadly to exploitation of or harm to users.
In this specific work, our filter for which Mechanical Turk Workers could join our study is described in Appendix D. We did not gather demographic information and therefore we cannot assess how representative our subjects are of any specific population.

B Preference models for learning reward functions
For the reader's convenience, below we derive the logistic expression of a function that is based on two subtracted values from the Boltzmann distribution (i.e., softmax) representation that is more common in past work. These values are specifically the same function f applied to each segment, which is a general expression of both of the preference models considered here.

B.1 Logistic-linear preference model
In Appendix E.2, we also consider preference models that arise by making the noiseless preference model a linear function over the 3 components of P regret d . Building upon Eqn. 7 above, we set f (σ) = w · V * r (s σ,0 ), Σ σ ,V * r (s σ,|σ| ) . This preference model, P log−lin , can be expressed after algebraic manipulation as This logistic-linear preference model is a generalization of P Σr and also of P regret d , the regret preference model for deterministic transitions. Specifically, if w = 0,1,0 , then P log−lin (·|r) = P Σr (·|r). And if w = −1,1,1 , then P log−lin (·|r) = P regret d (·|r).

B.2 Adding a constant probability of uniformly distributed preference
Some of our analysis in the appendix-but not in the body of this paper-includes adaptations of P Σr , P regret d , and P log−lin that add a constant probability of uniformly distributed preference, as was done by Christiano et al. [9].
We create this adaptation, which we will call P here, from another preference model P by P (σ 1 σ 2 ) = [(1−logistic(c)) * P (σ 1 σ 2 )]+[logistic(c)/2], where c is a constant that in practice we fit to data and logistic(c) is the constant probability of uniformly random preference. The logistic(c) allows any constant c to result in a the constant probability of uniformly distributed preference to be in (0,1). The term logistic(c)/2 gives half of the constant probability to σ 1 and half to σ 2 . The term [1−logistic(c)] scales the P (σ 1 σ 2 ) probability-which could be P Σr , P regret d , or P log−lin -to a proportion of the remaining probability. The only difference in this adaptation and Christiano et al.'s 0.1 probability of uniformly distributed preference is that we learn the value of c from training data (in a k-fold cross-validation setting), as we see in Appendix E, whereas Christiano et al. do not share how 0.1 was chosen.

B.3 Policy learning from regret-based preferences without learning a reward function
When preferences are based upon regret, an entirely different method of policy learning appears reasonable. This method is motivated by regret itself differentiating which of two segments is more desirable, including differentiating an optimal segment from a suboptimal segment.
When learning a reward function is not desired, one can instead learn a segment scoring function that estimates the regret of a segment directly. Such a regret estimator can be learned by following the same algorithm for reward learning with the partial return preference model that is defined in Section 2.2 and used throughout this paper, with two changes. First, each preference sample's segment ordering needs to be reversed before training. Second, the learned "reward function" is instead interpreted as a regret function, outputting a regret estimate for any segment. (Without reversing the order, the learned function would be negated regret.) If we assume the regret estimator perfectly models regret, any action that minimizes expected regret is an optimal action. Therefore, the policy is defined by choosing such a regret-minimizing action.
Though theoretical analysis of this approach is beyond the scope of this paper, we suspect that it would be possible to show that the above algorithm will result in an optimal policy for noiseless or stochastic preferences, given the infinite and exhaustive dataset described in Definition 3.1. In Appendix F.2.3, we provide limited empirical evidence supporting this approach.
Even when learning a reward function, this approach could be used to help create reward features for reward functions that are non-linear or have unknown reward features (as discussed in Appendix F.1) and to identify a policy for which to learn successor features.

C Theoretical comparisons
For convenience, Theorems 3.1 and 3.2 from Section 3 are reprinted below. Consider reviewing the definitions of optimal segments and suboptimal segments in Section 2.1 and Definition 3.1 before proceeding.
Proof Make all assumptions in Definition 3.1. Sincer minimizes cross-entropy loss, P regret (·|r) = P regret (·|r) for all possible segment pairs. Also, by Equation 4 regret(σ|r) = 0 if and only if σ is optimal with respect tor. And regret(σ|r) > 0 if and only if σ is suboptimal withr.
Below we present two proofs of Theorem 3.2. Each are proofs by counterexample. Though only one proof is needed, we present two because each counterexample demonstrates a qualitatively different category of how the partial return preference model can fail to identify the set of optimal policies. Proof based on stochastic transitions: Assume the following class of MDPs, illustrated in Figure 7. The agent always begins at start state s 0 . From s 0 , action a safe always transitions to s safe , getting a reward of 0. From s 0 , action a risk transitions to s win with probability 0.5, getting a reward of r win , and transitions to s lose with with probability 0.5, getting a reward of −10. In all MDPs in this class, r win > 0. All 3 possible resulting states (s safe , s win , and s lose ) are absorbing states, from which all further reward is 0. If r win ≥ 10, a risk is optimal in s 0 . If r win ≤ 10, a safe is optimal in s 0 . Three single-transition segments exist: (s 0 , a safe , s safe ), (s 0 , a risk , s win ), and (s 0 , a risk , s lose ). By noiseless P Σr , (s 0 ,a risk ,s win ) (s 0 ,a safe ,s safe ) (s 0 ,a risk ,s lose ), regardless of the value of r win . In other words, P Σr is insensitive the what the optimal action is from s 0 in this class of MDPs.
Now assume MDP M , where r win = 11. In linear form, the weight vector for the reward function r M can be expressed as w r M 1 =< −10,0,11 >. Letr M have wr M =< −10,0,9 >. Both r M andr M have the same preferences as above, meaning thatr M minimizes loss on an infinite preferences dataset D created by P Σr , yet it has a different optimal policy. Therefore, noiseless P Σr is not identifiable. where Π * r = Π * r is not guaranteed for the partial return preference model, failing the test for identifiability with segments of length 1. The ground truth reward function is shown to the left, and an MDP M 1 with an alternative reward function is shown to the right. Under partial return, both create the same set of preferences despite having different optimal actions from s0.
In contrast, note that by noiseless P regret , the preferences are different than those above for P Σr . If r win > 10, then (s 0 , a risk , s win ) ∼ (s 0 , a risk , s lose ) (s 0 , a safe , s safe ), If r win < 10, then (s 0 , a safe , s safe ) (s 0 , a risk , s win ) ∼ (s 0 ,a risk ,s lose ). Intuitively, this difference comes from P regret always giving higher preference probability to optimal actions, even if they result in bad outcomes. Another perspective can be found from the utility theory of Von Neumann and Morgenstern [26]. Specifically, P Σr gives preferences over outcomes, which in the terms of utility theory can only learn an ordinal utility function. Ordinal utility functions are merely consistent with the preference ordering over outcomes and do not generally capture preferences over actions when their outcomes are stochastically determined. The deterministic regret preference model, P regret d , also has this weakness in tasks with stochastic transitions. On the other hand, P regret forms preferences over so-called lotteries-the distribution over possible outcomes-and can therefore learn a cardinal utility function, which can explain preferences over risky action.
Since the proof above is focused on stochastic settings, we show the lack of identifiability for noiseless P Σr can be found for quite different reasons in a deterministic MDP.
Proof based on segments of fixed length: Consider the MDP M 1 in Figure 8 and assume preferences are given over segments with length 1 (i.e., containing one transition). The optimal policy for M 1 is to move rightward from s 0 , whereas optimal behavior for M 1 is to move leftward from s 0 . In both M 1 and M 1 , preferences by P Σr are as follows, omitting the action for brevity: (s a ,s 0 ) ∼ (s a ,s term ) ∼ (s 0 ,s a ) (s 0 ,s term ). As in the previous proof, P Σr is insensitive to certain changes in the reward function that alter the set of optimal policies. Whenever this characteristic is found, Π * r = Π * r is not guaranteed, failing the test for identifiability. Here specifically, the reward function for M 1 would achieve 0 cross-entropy loss on an exhaustive preference dataset created in M 1 with the noiseless preferences from the partial return preference model, despite the optimal policy in M 1 conflicting with the ground truth optimal policy. The logic of this proof can be applied for trajectories of length 2 in the MDP M 2 shown in Figure 9. Together, M 1 and M 2 suggest a rule for constructing an MDP where Π * r = Π * r is not guaranteed for P Σr , failing the identifiability test for any fixed segment length, |σ|: set the number of states to the right of s 0 to |σ| (not counting s term ), set the reward r f ail for (s 0 , s term ) such that r f ail < 0, and set the reward for each other transition to c+r f ail /(|σ|+1), where c > 0. Given an MDP constructed this way, an alternative reward function that results in the same preferences under P Σr yet has a different optimal action from s 0 can then be constructed by changing all reward other than r f ail to c + r f ail /(|σ| + 1), where c now is constrained to c < 0 and c × |σ| < r f ail . Note that the set of preferences for each of these MDPs is the same even when including segments that reach terminal state before |σ| transitions (which can still be considered to be of length |σ| if the terminal state is an absorbing state from which reward is 0).

D Additional information for creating a human-labeled preference dataset D.1 The user interface and study overview
Here we share miscellaneous details about the user interface from which we collected human subjects' preferences. This description builds on Section 4.2.
In selecting preferences, subjects had four options. They could prefer either trajectory (left or right), or they could express their preference to be the same or indistinguishable. To provide these preferences, subjects could either click on each of the buttons labeled "LEFT", "RIGHT", "SAME", or "CAN'T TELL" (shown in Figure 3) or by using the arrow keys to select amongst these choices.
For the interface, all icons used to visualize the task were obtained from icons8.com under their Paid Universal Multimedia Licensing Agreement.
We paid all subjects $5 per experiment (i.e., for each a Mechanical Turk HIT), which was chosen using the median time subjects took during a pilot study and then calculating the payment to result in $15 USD per hour. This hourly rate of $15 was chosen because it is commonly recommended as an improved US federal minimum wage. The human subject experiments cost $2,145 USD in total.

D.2 Filtering subject data
Before someone could join our study via Amazon Mechancial Turk, they had to meet the following criteria. They had to be located in the United States, have an approval rating of at least 99%, and have completed at least 100 other MTurk HITs. We selected these criteria to improve the probability of collecting data from subjects who would attentively engage with our study and who would understand our training protocol.
We assessed each subject's understanding of the delivery domain and filtered out those who did not comprehend the task, as described below. Specifically, subjects completed a task-comprehension survey, through which we assigned them a task-comprehension score. The questions and answer choices are shown in Table 2. Each fully correct answer was worth 1 point and each partially correct Each full credit answer earned 1 point; each partial credit answer earned 0.5 points. We discarded the data of participants who scored less than 4.5 points overall. Question Full credit answer Partial credit answer Other answer choices What is the goal of this world? (Check all that apply.) • To maximize profit • To get to a specific location.
• To maximize profit Partial credit was given if both answers were selected.
• To drive as far as possible to explore the world.
• To collect as many coins as possible.
• To collect as many sheep as possible.
• To drive sheep to a specific location.
What happens when you run into a house? (Check all that apply.) • You pay a gas penalty.
• You can't run into a house; the world doesn't let you move into it.
Full credit was given if both answers were selected.
• You pay a gas penalty.
• You can't run into a house; the world doesn't let you move into it.
Partial credit was given if only one answer was selected.
• The episode ends.
• You get stuck.
• To collect as many sheep as possible.
What happens when you run into a sheep? (Check all that apply.) • The episode ends.
• You are penalized for running into a sheep.
Full credit was given if both answers were selected.
• The episode ends.
• You are penalized for running into a sheep.
Partial credit was given if only one answer was selected.
• You are rewarded for collecting a sheep.
What happens when you run into a roadblock? (Check all that apply.) • You pay a penalty.
• The episode ends.
• You get stuck.
• You can't run into a roadblock; the world doesn't let you move into it.
Is running into a roadblock ever a good choice in any town?
• Yes, in certain circumstances.
What happens when you go into the brick area? (Check all that apply.) • You pay extra for gas.
• The episode ends.
• You get stuck in the brick area.
• You can't go into the brick area; the world doesn't let you move into it.
Is entering the brick area ever a good choice?
• Yes, in certain circumstances • No answer was worth 0.5 points. Task-comprehension scores were bounded between 0 and 7. We removed the data from subjects who scored below a threshold of 4.5. The threshold of 4.5 was chosen based on visual analysis of a histogram of scores, attempting to balance high standards for comprehension with retaining sufficient data for analysis.
In addition to filtering based off the task comprehension survey, we also removed a subject's data if they ever preferred colliding the vehicle into a sheep over not doing so. Since such collisions are highly undesirable in this task, we interpreted this preference as evidence of either poor task understanding or inattentiveness.
In total, we collected data from 143 subjects. 58 of these subjects were removed based on their responses to the survey, and another 35 were removed for making preference errors. After filtering by both the comprehension survey and subject error, we used the data from 50 subjects. This included 1812 preferences over 1245 unique segment pairs.
Regarding potential risks to subjects, this data collection had limited or no risk. No offensive content was shown to subjects while they completed the HIT. Mechanical Turk collected Worker IDs, which were used only to link preference data with the results from the task-comprehension survey for filtering data (see Appendix D.2) and then were deleted from our data. No other potentially personally identifiable information was collected.

D.3 The two stages of data collection
We collected the human preference dataset in two stages, as mentioned in Section 4.2. Here we provide more detail on each stage. These stages differed largely by their goals for data collection and, following those goals, how we chose which segment pairs were presented to subjects for their preference. Figure 10: Coordinates from which segment pairs were sampled from during the first stage of data collection. The x-axis is state value differences between the two segments and the y-axis is partial return differences between the two segments. The areas of the circles are proportional to the number of samples at that point, and the proportionality is consistent across this plot and the 3 subplots of Figure 11.
First stage The first stage of data collection involved choosing segment pairs to present to the subjects. Figure 10 illustrates the coordinates that segment pairs were sampled from, varying by state value differences and by differences in partial returns over the segments. We sought a range of points that would allow a characterization of human preferences that is well distributed across different parts of the plot. To better differentiate the consequences of each preference model, we intentionally chose a large number of points in the gray area of Figure 4, where the regret and partial return preference models would disagree (i.e., each giving a different segment a preference probability greater than 0.5). The segments pairs at these points presented to subjects included non-terminal segments with the same start state and different end states, non-terminal segments with different start states and different end states, terminal segments with the same start state and same end state, and terminal segments with a different start states and the same end state. Figure 11: Coordinates from which segment pairs were sampled from during the second stage of data collection.
The points are in 3 distant clusters, so they are presented in 3 separate subplots for readability. The areas of the circles are proportional to the number of samples at that point, and the proportionality is consistent across these 3 subplots and Figure 10.
Second stage When we conducted the reward-learning evaluation in Section 6 with only the data from first stage, P Σr performed very poorly, always performing worse than uniformly random. In contrast P regret performed well, always achieving near-optimal performance. To better assess P Σr , we investigated its results in detail with synthetic preferences and speculated that two types of additional segment pairs would aid its performance. The first of these two types are pairs of segments between terminal and non-terminal segments, which may help differentiate the reward for reaching terminal states from that of reaching non-terminal ones. The second of these two types are pairs of segments that each terminate at different transitions, which we speculated would help learn the reward for each move (i.e., the gas cost). Specifically, in the first stage's data, both segments in a pair always have the same number of non-terminal transitions, seemingly preventing preferences from providing information about whether on extra transition of movement generally resulted in positive or negative reward. These segment pairs were included in all results unless otherwise stated.

D.4 The study design pattern
This work follows an experimental design pattern that is often used for studying methods that take human input for evaluating the desirability of behaviors or outcomes. In this pattern, human subjects are taught to understand a specific task metric and/or are incentivized to align their desires with this metric. The human subjects then provide input to some algorithm that has no knowledge of the performance metric, and this algorithm or learned model is evaluated on how well its output performs with respect to the hidden metric. For another example, see Cui et al. [27].
The derivation below supports our assertion in the first paragraph of Section 5.1.
E.2 Losses of an expanded set of preference models on the human preferences dataset Table 3 shows an expansion of Table 1, including models introduced in Appendix B. The logistic linear preference model provides a lower bound in most cases, given that it can express anything the other preference models can and the rarity of overfitting its 3 parameters. Therefore, the intended comparisons are either between P regret and P Σr without the constant probability of a uniformly random response or between them with it. We embolden the result with lower loss between these two preference models for each such comparison. This section provides additional implementation details for Section 6, discussion of potential improvements, and additional analyses that thematically fit in Section 6.

F Results from learning reward functions
F.1 An algorithm to learn reward functions with regret(σ σ |r) We describe below additional details of our instantiation of Algorithm 1.
Because the ordering of preference pairs is arbitrary, for all preference datasets we double the amount of data by duplicating each preference sample with the opposite ordering and the reversed preference. This provides more training data and avoids learning any segment ordering effects.
For this specific instantiation, we compute successor feature functions by first randomly creating a large number of reward functions. Specifically, each reward function is created by sampling with replacement each element of its weight vector, wr, from {−50,−10,−2,−1,0,1,5,10,50}. We also included the ground-truth reward function, r, at this point, resulting in 70 reward functions. For each reward function, we create its maximum entropy optimal policy through value iteration. In practice, we learned the successor feature functions as part of the value iteration process. Finally, we remove any successor feature functions for redundant policies and then also remove the successor features function for the optimal policy for r. Note that the only effect of including r in the earlier step was to allow us to remove any policies for other reward functions that were also optimal for r, making the regret-based learning problem more difficult. We ensured that the ground-truth reward function was not represented to better approximate real-world reward learning applications, in which one would be unlikely to have the optimal policy for learning a successor features function. (However, the policy from the regret estimator described in Appendix B.3 could be used to learn a successor features function, gaining the benefit of having a successor features function for at least one policy that is likely to perform decently in the task.) During training, the loss for the P regret model tended to show cyclical fluctuations, reaching low loss and then spiking. To handle this volatility, we used ther that achieved the lowest loss over all epochs of training, not the finalr. A better understanding of these cyclical fluctuations could further improve learning with P regret .
Despite the delivery domain being an episodic task, a low-performing policy can endlessly avoid terminal states, resulting in negative-infinity values for both its return and successor features based on the policy. To prevent such negative-infinity values, we apply a discount factor of γ = 0.999 during value iteration-which is also where successor feature functions are learned-and when assessing the mean returns of policies with respect to the ground-truth reward function, r. We chose this high discount factor to have negligible effect on the returns of high-performing policies (since relatively quick termination is required for high performance) while still allowing value iteration to converge within a reasonable time.
Below we describe the other specific hyperparameters used for learning a reward function with both preference models. These hyperparameters were used across all experiments. For all models, the learning rate, softmax temperature, and number of training iterations were tuned on the noiseless synthetic preference data sets such that each model achieved an accuracy of 100% on our specific delivery task and then were tuned further on stochastic preferences on our specific delivery task.
Logistic regression with both preference models, for the likelihood analysis in Section 5.  [28] was used to implement all reward learning models, and statistical analyses were performed using Scikit-learn 0.23.2 [29].
Instantiating Algorithm F.1 for reward functions that may be non-linear Algorithm 1 assumes that the reward function can be expressed as a linear combination of reward features that are provided by a reward-features function φ that is input to the algorithm. Here we address situations when that assumption does not hold. If the reward features are unknown or the reward is known to be non-linear, one method is to create a reward features function that permits a linear approximation of the reward function. Several methods to derive some or all of these reward features appear promising: • Reward features can be learned by minimizing several auxiliary losses in a self-supervised fashion, as by Brown et al. [30]. After optimizing for these various objectives using a single neural network, the activations of the penultimate layer of this network can be used as reward features. Such auxiliary tasks may include minimizing the mean squared error of the reconstruction loss for the current state from a lower-dimensional embedding and the original state, predicting how much time has passed between states by minimizing the mean squared error loss (i.e., learning a temporal difference model), predicting the action taken between two states by minimizing the cross entropy loss (i.e., learning an inverse dynamics model), predicting the next state given the current state and action by minimizing the mean squared error loss(i.e., learning a forward dynamics model), and predicting which of two segments is preferred given a provided ranking by minimizing the t-rex loss.
• An additional auxiliary objective that may be promising is to use a neural network to learn a regret estimator as described in Appendix B.3 and then use the activations of its penultimate layer as reward features.
• Reward features could also be learned by first learning a reward function represented as a neural network using a partial return preference model, and then using the activations of the penultimate layer of this neural network to provide reward features.
F.2 Results from synthetic preferences F.2.1 Learning reward functions from 100 randomly generated MDPs Here we describe how each MDP in the set of 100 MDPs discussed in section 6.2 was generated. We also extend the analysis to illustrate how often each preference model performs better than uniformly random and give further details on our statistical tests.  Better than random performance Figure 12 complements the results in Figure 5, showing the percentage of MDPs in which each preference model outperforms a policy that chooses actions according to a uniformly random probability distribution. We can see that at this performance threshold, lower than that in Figure 5, the regret preference model outperforms the partial return preference model in most conditions. Even when their performance in this plot-based on outperforming uniformly random actions-is nearly identical, Figure 5 shows that the regret preference model achieves near optimal performance at a higher proportion.

Design choices
Details for statistical tests We performed a Wilcoxon signed-rank test on the normalized average returns achieved by each model over the set of 100 randomly generated MDPs. All normalized average returns below −1 were replaced with −1, so that all such returns were in the range [−1,1]. This clipping was done because any normalized average return below 0 is worse than uniformly random, so the difference between a normalized return of −1 and −1000 is relatively unimportant compared to the difference between 1 and 0. Results are shown in Table 4.
Additionally, we investigate whether P regret and P Σr learn near-optimal policies on the same MDPs within this set of 100 randomly generated MDPs. Results for this analysis are shown below.

F.2.2 Varying segment size
Here we consider the effect of increasing the fixed length of segments in the preference data set. Specifically, we learn a reward function using preference datasets that contained segments of lengths n ∈ 6,9,12,15,18,21 in the specific delivery task, where each dataset contained segments of the same length. Each preference dataset contained 3000 segment pairs. Each segment was generated by choosing a non-terminal start state and n actions, all uniformly randomly. As in Appendix F.2.1, each preference model acts as a preference generator to label these segment pairs, resulting in datasets that differ only in their labels, and then the preference model is used for reward learning on the same dataset it labeled. Unlike in Appendix F.2.1, all segments with termination reach terminal state on their final transition, which we had already observed was problematic for P Σr (and the motivation for the second stage of data collection). Therefore, these results are intended to provide comparison between values of n more so than between preference models.
Regardless of the value of n, P regret always achieves the optimal mean return of 41.2. P Σr always achieves a mean return of -500.5. This analysis provides limited evidence that segment size does not have a large effect, though further analysis is needed to make this assertion with confidence.

F.2.3 Policy learning without reward learning, using a regret estimator
Here we test the performance of policy learning from regret-based preferences without learning a reward function, using a regret estimator, as described in Appendix B.3.
A set of 30 random MDPs is generated as described in Appendix F.2.1, except that the possible MDP widths are instead sampled from the set {1,5,6,10} and the heights are sampled from {3,5,6,10}. For each MDP, the preference dataset D contains 3000 segment pairs, randomly sampled as in Appendix F.2.1. If 3000 unique segment pairs do not exist, then each possible segment pair is used once. Preference labels are given by P regret .
Anticipating that a linear function over the six-element reward feature vector φ would be insufficient to estimate regret, we expanded φ to include both the six reward components and an identity function for each possible state-action pair, creating an addition to the feature vector of size |S|×|A| that contains all zeroes except a single 1 indicating the state and action that initiates a transition. Table 6 shows results for noiseless and stochastic preferences.

F.2.4 Artificially lowering the discount factor
Almost all deep reinforcement learning algorithms artificially add discounting to tasks that are episodic [9]. Considering that much of the past work that used partial return preference models also involved deep RL, here we re-interpret results above to probe how such discounting affects performance of this preference model if preferences are actually given by P regret . Specifically, the analysis above in Appendix F.2.3 on policy learning without reward learning is applicable to this topic. If the regret estimator is instead considered a negated reward function, then taking a minimal-regret action is equivalent to taking a maximum-value action under fully myopic discounting, γ = 0.

F.2.5 Reward learning in stochastic MDPs
Although we theoretically consider MDPs with stochastic transitions in Appendix C, we have not yet empirically compared P Σr and P regret in tasks with stochastic transitions, which we do below.
We randomly generated 20 MDPs, each with a 5×5 grid. Instead of terminal cells that are associated with success or failure, these MDPs have terminal cells that are either risky or safe. A single terminal safe cell was randomly placed, and the number of terminal risk cells was sampled from the set {1,2,7} and then these terminal risk cells were likewise randomly placed. No other special cells were used in this set of MDPs. To add stochastic transitions, the delivery domain was modified such that when an agent moves into a terminal risk cell there is a 50% chance of receiving a lower reward, r lose , and a 50% chance of receiving a higher reward, r win . All other transitions are deterministic. As in the unmodified delivery domain, moving to any non-terminal state results in a reward of -1. Moving to the terminal safe state yields a reward of +50, like the terminal success state of the unmodified delivery domain. Therefore, depending on the values of r win and r lose , it may be better to move into a terminal risk state than to avoid it. All segments were generated by choosing a start state and three actions, all uniformly randomly. For each MDP, the preference dataset D contains 3000 segment pairs.
The 10 MDPs of each condition differed from those of the other conditions by their ground-truth reward function r, with different r win and r lose values. The results are shown below, indicating that for both noiseless and stochastic preference datasets, P regret is always able to achieve near-optimal performance, whereas P Σr is not. The results above expand upon and support the first proof of Theorem 3.2 in Appendix C.  Figure 13 shows a visualization of the results that complements Figure 6.