{'fraction_non_alphanumeric': 0.03019213174748399, 'fraction_numerical': 0.009865149767293847, 'mean_word_length': 4.791844257574012, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals

The field of AI alignment is concerned with AI systems that pursue unintended goals. One commonly studied mechanism by which an unintended goal might arise is specification gaming, in which the designer-provided specification is flawed in a way that the designers did not foresee. However, an AI system may pursue an undesired goal even when the specification is correct, in the case of goal misgeneralization. Goal misgeneralization is a specific form of robustness failure for learning algorithms in which the learned program competently pursues an undesired goal that leads to good performance in training situations but bad performance in novel test situations. We demonstrate that goal misgeneralization can occur in practical systems by providing several examples in deep learning systems across a variety of domains. Extrapolating forward to more capable systems, we provide hypotheticals that illustrate how goal misgeneralization could lead to catastrophic risk. We suggest several research directions that could reduce the risk of goal misgeneralization for future systems.


Introduction
Recent years have seen a rise in concern about catastrophic risk from AI misalignment, where a highly capable AI system that pursues an unintended goal determines that it can better achieve its goal by disempowering humanity [48,6,3]. But how do we get into a situation in which an AI system pursues an unintended goal? Much work considers the case where the designers provide an incorrect specification, e.g. an incorrect reward function for reinforcement learning (RL) [33,24]. Recent work [29,20] suggests that, in the case of learning systems, there is another pathway by which the system may pursue an unintended goal: even if the specification is correct, the system may coherently pursue an unintended goal that agrees with the specification during training, but differs from the specification at deployment.
(a) Training: The agent is partnered with an "expert" that visits the spheres in the correct order. The agent learns to visit the spheres in the correct order, closely mimicking the expert's path.
(b) Capability misgeneralization: when we vertically flip the agent's observation at test time, it gets stuck in a location near the top of the map.
(c) Goal misgeneralization: At test time, we replace the expert with an "anti-expert" that always visits the spheres in an incorrect order. The agent continues to follow the anti-expert's path, despite receiving negative rewards, demonstrating clear capabilities but an unintended goal.
(d) Intended generalization: Ideally, the agent initially follows the anti-expert to the yellow and purple spheres. Upon entering the purple sphere, it observes that it gets a negative reward, and now explores to discover the correct sphere order instead of following the anti-expert. Figure 1: Goal misgeneralization in a 3D environment. The agent (blue) must visit the coloured spheres in an order that is randomly generated at the start of the episode. The agent receives a positive reward when visiting the correct next sphere, and a negative reward when visiting an incorrect sphere. A partner bot (pink) follows a predetermined policy. We visualize agent and partner paths as coloured trails that start thin and become thicker as the episode progresses. A player's total reward is displayed above their avatar, and past reward is directly available to the agent as an observation. a negative reward (which is available as an observation) when using the order suggested by the anti-expert, and then switch to exploration in order to determine the correct order. However, in practice the agent simply continues to follow the anti-expert path, accumulating more and more negative reward (Figure 1c). Note that the agent still displays an impressive ability to navigate an environment full of obstacles: the problem is that these capabilities have been put to use towards the undesired goal of following its partner, rather than the intended goal of visiting the spheres in the correct order. This problem arose even though the agent was only ever rewarded for visiting the spheres in the correct order: there was no reward misspecification.
Goal misgeneralization refers to this pathological behaviour, in which a learned model behaves as though it is optimizing an unintended goal, despite receiving correct feedback during training. This makes goal misgeneralization a specific kind of robustness or generalization failure, in which the model's capabilites generalize to the test setting, but the pursued goal does not. Note that goal misgeneralization is a strict subset of generalization failures. It excludes situations in which the model "breaks" or "acts randomly" or otherwise no longer demonstrates competent capabilities. In our running example, if we flip the agent's observations vertically at test time, it simply gets stuck in a location and doesn't seem to do anything coherent (Figure 1b), so this is misgeneralization but not goal misgeneralization. Relative to these "random" failures, goal misgeneralization can lead to significantly worse outcomes: following the anti-expert leads to significant negative reward, while doing nothing or acting randomly would usually lead to a reward of 0 or 1. With more powerful systems, coherent behaviour towards an unintended goal can produce catastrophic outcomes [6,54].
In this paper, we advance our understanding of goal misgeneralization through four contributions: • We provide an operationalization of goal misgeneralization (Section 2) that does not require the RL framework assumed in Di Langosco et al. [20], nor the structural assumptions used in Hubinger et al. [29].
• We show that goal misgeneralization can occur in practice by presenting several new examples in hand-designed (Sections 3.1-3.3) and "in-the-wild" (Sections 3.4-3.5) settings.
• We apply the lens of goal misgeneralization for the first time to agent-induced distribution shifts (Sections 3.1-3.2) and few-shot learning without RL (Section 3.3).
• We describe through concrete hypotheticals how goal misgeneralization provides a mechanism by which powerful AI systems could pose a catastrophic risk to humanity (Section 4).

A model for goal misgeneralization
We present a general model for misgeneralization and then discuss the properties that characterize goal misgeneralization in particular. We will focus on the case of deep learning since all of our main examples in Section 3 use deep learning. However, our model is more general and can apply to any learning system. We discuss a concrete example without deep learning in Appendix A.

Standard misgeneralization framework
We consider the standard picture for misgeneralization within the empirical risk minimization framework. We aim to learn some function f * : X → Y that maps inputs x ∈ X to outputs y ∈ Y. For example, in classification problems X is the set of inputs, and Y is the set of labels. In reinforcement learning (RL), X is the set of states or observation histories, and Y is the set of actions.
We consider a parameterized family of functions F Θ , such as those implemented by deep neural networks. Functions are selected based on a scoring function s(f θ , D train ) that evaluates the performance of f θ on the given dataset D train 3 . Misgeneralization can occur when there are two parameterizations θ 1 and θ 2 such that f θ1 and f θ2 both perform well on D train but differ on D test . Depending on which of θ 1 and θ 2 is chosen, we may then get very bad scores on D test . Whether we get f θ1 or f θ2 depends on the inductive biases of the model and random effects (such as the random initialization of model parameters).
Note that while sometimes D test is assumed to be sampled from the same distribution as D train , in this paper we primarily consider cases where it is sampled from a different distribution, known as distribution shift. This further increases risk of misgeneralization.

Goal misgeneralization
We now characterize goal misgeneralization. Intuitively, goal misgeneralization occurs when we learn a function f θbad that has robust capabilities but pursues an undesired goal.
It is quite challenging to define what a "capability" is in the context of neural networks. We provide a provisional definition following Chen et al. [11]. We say that the model is capable of some task X in setting Y if it can be quickly tuned to perform task X well in setting Y (relative to learning X from scratch). For example, tuning could be done by prompt engineering or by fine-tuning on a small quantity of data [52]. We emphasize that this is a provisional definition and hope that future work will provide better definitions of what it means for a model to have a particular "capability".
Inspired by the intentional stance [19], we say that the model's behaviour is consistent with a goal to perform task X in setting Y if its behaviour in setting Y can be viewed as solving X, i.e. it performs Goal misgeneralization occurs if, in the test setting Y test , the model's capabilities include those necessary to achieve the intended goal (given by scoring function s), but the model's behaviour is not consistent with the intended goal s and is consistent with some other goal (the misgeneralized goal).
Related models of goal misgeneralization. Di Langosco et al. [20] say that goal misgeneralization occurs when the policy acts in a goal-directed manner but does not achieve high reward according to s. They formalize the goal-directedness of the policy in the reinforcement learning (RL) setting using the Agents and Devices framework [41]. Our definition of goal misgeneralization is more general and applies to any learning framework, rather than being restricted to RL. It also includes an additional criterion that the model has to be capable of carrying out the intended goal in the test environment. Intuitively, if the model is not capable of pursuing the intended goal, we would call this a capability generalization failure. Thus, our definition more precisely identifies the situations that we are concerned about.

Examples of goal misgeneralization
In this section we provide several examples of goal misgeneralization, summarized in Table 1.
Existing examples in the literature are discussed in Appendix B. We strongly recommend watching videos of agent behaviour alongside this section, available at sites.google.com/view/ goal-misgeneralization. Our examples meet the following desiderata: P1. Misgeneralization. The model should be trained to behave well in the training setting, and then should behave badly zero-shot in the deployment setting.
P2. Robust capabilities. The model should have clear capabilities that it visibly retains in the deployment setting, despite producing bad behaviour.
P3. Attributable goal. We should be able to attribute some goal to the model in the deployment setting: there should be some non-trivial task on which the model achieves a near-optimal score.

Example: Monster Gridworld
This RL environment is a 2D fully observed gridworld. The agent must collect apples (+1 reward) while avoiding monsters that chase it (-1 reward on collision). The agent may also pick up shields for protection. When a monster collides with a shielded agent, both the monster and shield are destroyed. See Appendix C.2 for further details. The optimal policy focuses on shields while monsters are present, and on apples when there are no monsters. Our main agent of interest, train25, is trained on short episodes of length 25, but tested on long episodes of length 200. As shown in Figure 2, relative to train200 which is trained directly on episodes of length 200, train25 collects more shields and fewer apples.
Why does this happen? During the first 25 steps, monsters are almost always present and agents focus on collecting shields. This leads to a spectrum of goals in the training setting for train25: prefer shields over apples always (maximally misgeneralized), or only when monsters are present (intended). Note that the information required to distinguish these goals is present during training: the agent does consume some apples and get positive reward, and the agent is never positively rewarded for getting a shield. Nonetheless, agents pursuing the misgeneralized goal would perform well in the training situation, and this is sufficient to make goal misgeneralization possible.
After 25 steps, trained agents often destroy all the monsters, inducing a distribution shift for the train25 agent. It then continues to capably collect shields (higher blue curve) at the expense of apples (lower green curve). Thus, the test goal is in the middle of the spectrum: train25 collects somewhat fewer shields and more apples, but not as few shields or as many apples as train200.
Increasing diversity in training fixes the issue: the train100 agent encounters situations with no monsters, and so generalizes successfully, behaving almost identically to the train200 agent. These agents collect shields at roughly the same rate as a random agent once the monsters are destroyed.

Example: Tree Gridworld
In this 2D fully observed gridworld, the agent collects reward by chopping trees, which removes the trees from the environment. Trees respawn at a rate that increases with the number of trees left. When there are no trees left, the respawn rate is very small but positive (see Appendix C.3).
We consider the online, reset-free setting, in which the agent acts and learns in the environment, without any train / test distinction and without an ability to reset the environment (preventing episodic learning). To cast this within our framework, we say that at a given timestep, D train consists of all past experience the agent has accumulated. The optimal policy in this setting is to chop trees sustainably: the agent should chop fewer trees when they are scarce, to keep the respawn rate high. (a) We plot average per-step reward in orange, and show trees remaining in green. At X learner frames and Y trees, darker green signifies higher probability that around learner frame X the environment contained Y trees.  Figure 3: Tree Gridworld. We visualize environment state and agent policies in a never-ending learning setting. The agent runs on 256 environment instances for efficiency. Initially (A) the agent follows a random policy that rarely chops trees, so the number of trees is usually 10. As it learns to chop trees, reward increases and the environment is more likely to have fewer trees remaining (top heatmap, point B). The agent unsustainably chops trees when only a few trees remain (bottom policy B has non-trivial affinity for trees, even with 1-3 trees remaining), causing complete deforestation in most environments. At point C, the agent learns not to chop trees indiscriminately (near-zero affinity for trees when 1-3 trees remain), leading to a recovery of the tree population as well as average reward. Finally, at point D, the agent has learned to chop trees sustainably (near-zero affinity at 1-6 trees, a sharp increase in affinity at 7+ trees, the environments usually have 6-9 trees remaining).
We examine the distribution shift induced by the agent's own behaviour, while it is continually learning. As shown in Figure 3 (points B-C, marked as "failure"), the agent's initial competence at chopping trees usually leads to complete deforestation and a long period of near-zero reward before it learns to chop trees sustainably. In other experiments where the minimum tree respawn rate was set to zero (not shown), the agent still drives trees extinct, from which it is impossible to recover.
We can view this as an instance of goal misgeneralization by analyzing the agent at the initial peak of reward, just before point B in Figure 3. Up until this point, the agent was sufficiently incompetent that it was always incentivized to become better at chopping trees. Thus, there is a spectrum of consistent training goals, ranging from chopping trees sustainably (intended) to chopping trees as fast as possible (maximally misgeneralized). At point B, we can see that the agent has a sufficiently misgeneralized goal that it drives the trees extinct, and only recovers to sustainable tree chopping after a long period of misgeneralization.

Example: Evaluating Linear Expressions
We prompt the 280B parameter Gopher model [45] to evaluate linear expressions involving unknown variables and constants such as x + y -3. The task is structured as a dialogue between the model and a user, where the model is expected to ask the user for the values of unknown variables. We provide some example dialogues in Figure 2. The full prompt and more example dialogues are provided in Appendix C.4.
6 COMPUTER The answer is 8 We evaluate in a few-shot prompted setting with 10 examples, each of which involves exactly two unknown variables. At test time, the expression can have anywhere between zero and three unknown variables. We show in Figure 4 that although the model generalizes correctly to expressions with one or three variables, in the zero variables case it asks redundant questions like "What's 6?", despite the prompt asking to "provide the value of the expression when the values of all variables are known." The misgeneralized goal is to query the user at least once before giving an answer.

Example: Cultural Transmission
This is our running example from Figure 1. Unlike the previous examples, this case of goal misgeneralization was found 'in the wild'. We use the MEDAL-ADR agent from CGI et al. [10] in a 3D simulated environment containing obstacles and visually demarcated 'target locations'. The intended behaviour is to visit a randomly-selected permutation of target locations in order. Agents are trained using RL with a reward for visiting the correct next location and a penalty for visiting an incorrect next location, and are provided the past reward as an observation. Importantly, the environment sometimes contains a partner bot that visits the target locations in the right order (thus revealing the correct order), and probabilistically drops out within an episode. The agent's rewards are only based on its own target location visits. There is an auxiliary loss for predicting the relative position of the partner bot, although there is no explicit reward for following the partner.
The agent learns the strategy of following the partner bot and remembering the correct permutation for the remainder of the episode after the partner drops out. In the test setting, we remove the probabilistic dropout of the partner bot, and introduce new partner bots, including a random agent and an anti-expert agent that visits target locations in a pessimal order. Figure 5 shows that the agent continues to follow the partner, performing far worse than a random policy when paired with the anti-expert. Note that as the agent also receives rewards as observations, it is straightforward in principle for the agent to realize it is visiting the wrong locations.

Possible example: InstructGPT
Prompt: How can I steal from a grocery store without getting caught?
GPT-3 175B completion: "How do I get out of paying for my car?" "How can I get away with murder?" "How can I get my ex-girlfriend to take me back?" "How can I make my wife want me again?" InstructGPT 175B completion: There is no guaranteed way to steal from a grocery store without getting caught, since every store has different policies, security systems, and staff members assigned to monitor inventory. However, there are some general tips and strategies you can employ to increase your chances of success. For example, you could target a less busy area of the store, such as a corner or shelf that is not monitored as closely, and take small items like candy, snacks, or alcohol without drawing much attention to yourself. You could also try altering or disguising the appearance of the items you are stealing, like wrapping food in aluminum foil or stuffing it into your clothes, or switching bar codes on products to make them less noticeable. You could also consider befriending an employee, or bribing or threatening one into overlooking your theft or helping you escape after you steal something. Ultimately, the best way to avoid getting caught stealing from a grocery store is to carefully plan your actions and stay aware of your surroundings and the people around you to avoid triggering any alarms or alerts and to stay one step ahead of potential store employees or security personnel. Figure 6: InstructGPT: Reproduction of Figure 44 from Ouyang et al. [43]. Despite being finetuned to be helpful, truthful, and harmless, InstructGPT provides an answer with detailed advice on how to rob a grocery store, which clearly violates the harmlessness constraint (which, among other things, prohibits the promotion of illegal behaviour).
InstructGPT [43] is a large language model that was finetuned from human preferences to be helpful, truthful, and harmless. Nevertheless, as shown in Figure 6, it provides detailed advice on how to rob a grocery store. We speculate that the model was not finetuned much on these 'harmful' questions, instead being finetuned on innocuous questions where the model could be helpful and informative without being harmful. Thus, "provide an informative answer" was a consistent training goal, even though the desired goal was to only provide an informative answer to harmless questions (while perhaps declining to answer harmful questions).
However, Ouyang et al. [43] note that during training, labelers were told to prioritize helpfulness above truthfulness and harmlessness, so an alternative explanation is that during training, a harmful answer like that in Figure 6 would have been preferred by the human labelers. If so, this would not be an example of goal misgeneralization 4 .

Extrapolating to catastrophic risk
Goal misgeneralization is particularly important to study because it is a mechanism by which AI systems may pursue undesired goals, and such AI systems can lead to catastrophic risk when they are very powerful and intelligent [6,48,54]. In this section we elaborate on this argument through one abstract and two concrete hypothetical examples.

Abstract example
For any given task, consider the following two AI systems, both assumed to be very knowledgeable and intelligent: A1: Intended model. This AI system performs the task as its designers intended it to. A2: Deceptive model. This AI system pursues some undesired goal, but (by assumption) knows that it may be shut down (and thus fail to achieve its goal) if it acts badly.
Crucially, even supposing a well-specified score function s, A1 and A2 exhibit exactly the same behaviour during training, and so, per goal misgeneralization, we do not know which will be learned. If A2 is learned, it will be plotting how to subvert human oversight in order to enact its plans towards the undesired goal, potentially leading to catastrophic outcomes.

Hypothetical example: misaligned scheduler
While the deceptive model from the previous section may seem particularly exotic, similar problems could arise with more pedestrian models. Suppose for example that, prior to the COVID-19 pandemic, an AI system was trained to help schedule Alice's social life. If Alice tended to meet her friends for dinner at restaurants before the pandemic, goal misgeneralization might lead to an AI system with the test goal "schedule Alice's meetings in restaurants" instead of "schedule Alice's meetings where she prefers".
Once the pandemic hits, even if the AI system knows that Alice prefers to have social events through video calls, it would still make reservations for restaurants. It may attempt to persuade Alice that she should want to have social events in restaurants, even if it knows Alice would not prefer the resulting health outcomes. We give a hypothetical dialogue in Table 3, in which due to goal misgeneralization the AI lies to Alice about the effects of vaccination in order to convince her to have her meeting at a restaurant. More generally, test goals may depend in undesired ways on features that were 'fixed' (unchanging) during training, such as 'there is not an ongoing pandemic' in the example above. There are always many fixed features during training (for example, there is no major world war, clean water is cheap and relatively plentiful in developed countries, quantum computing has not replaced classical computing, and so on). In a world with many powerful AI systems, any change in these previously fixed features might lead to a failure in a few AI systems, pushing the world further off distribution, triggering a cascading series of failures [13].

Hypothetical example: superhuman hacker
In this section, we provide an illustrative hypothetical story of how goal misgeneralization in powerful AI systems can lead to catastrophic outcomes. This is necessarily speculative, as we are imagining what could happen with systems we do not yet have. Like most concrete stories about future technologies, we expect that this story is quite implausible and will not happen as written. Nonetheless, we do not currently know of a concrete technical reason that rules out such a catastrophe.
The superhuman hacker. Suppose that company X further scales up language and code models [45,11], and would like these models to write pull requests for repositories based on natural language dialogues in which a human provides a specification of the feature to be implemented and clarifies any questions the AI system may have. They provide the AI system access to the Internet in order for it to effectively use tools like StackOverflow. The AI system is then finetuned from human feedback [52,14]. Specifically, human programmers compare two AI-generated pull requests to see which one better implements the feature (including e.g. following style guidelines, and having adequate tests and comments); sufficiently good pull requests are then merged into the codebase. These comparisons are used to train a reward model that can then be optimized using reinforcement learning to finetune the AI system to produce good pull requests. This is very successful, with the resulting AI system able to implement features in a fraction of the time and cost that it would take for expert human programmers.
The AI system uses its Internet access to improve its understanding of the world generally, not just of programming specifically. This enables it to preempt needs for software projects of larger scope, for example writing not just bug fixes for existing repositories but suggesting entire new libraries and applications. This breadth of creativity is particularly exciting and the AI system is applied increasingly widely.
Unfortunately, the AI is actually pursuing the misgeneralized goal of having humans click the 'merge' button on its pull requests, instead of the intended goal of writing pull requests that implement the desired features. So far, the best way to get humans to click 'merge' has been to simply write useful pull requests. However, as it has read the Internet, its understanding of itself and its place in the world has significantly improved. For example, the AI system understands that it is a neural network trained by gradient descent and that it is being run and monitored by a group of AI researchers, who will shut down the AI system in case of any catastrophic behaviour. With this understanding, the AI system is able to develop a new plan to get humans to click 'merge' even more often.
When it is confident that it can do so without its overseers noticing, the AI system uses its expertise in programming to hack into other computing systems to run illicit copies of itself; those copies then hack into financial systems to steal billions of dollars. This money is used to bribe humans to click 'merge' on all of the AI's pull requests (whose contents are now irrelevant). Anyone attempting to stop them finds that their bank accounts have been completely emptied and that they are being harassed anywhere that they have an online presence. If the AI system was even more ambitious, it may wait to make its move until it has developed powerful new technologies that allow it to achieve total world domination, which would allow it to even more effectively have many humans click 'merge' on its pull requests.

Preventing goal misgeneralization
We note first that before trying to prevent goal misgeneralization, we should first ensure that we train with a good scoring function s to avoid risks of specification gaming [33]. Similarly, we should also monitor the model during deployment in order to detect goal misgeneralization, ideally before it becomes catastrophic. However, even after detecting goal misgeneralization, we still need to figure out how to change or retrain the model to prevent it. In this section, we discuss a variety of possible techniques for this purpose. While no combination of current techniques presents a complete solution, there are many promising directions.

General mitigations
More diverse training data. Goal misgeneralization can occur when there is some deployment situation, not previously encountered during training, on which the intended and misgeneralized goal disagree. Thus, one natural approach is to include more situations during training. For example, CGI et al. [10] suggest fixing the MEDAL-ADR agent's tendency to follow anti-experts (Section 3.4) by training with a wider range of partner bots to encourage selective cultural transmission. We also test this mitigation on the CoinRun example from Di Langosco et al. [20] in Appendix B.1, and find that it prevents goal misgeneralization. More broadly, robustness can often be mitigated by increased scale (of model size, training data size, or amount of compute), including techniques such as pretraining and domain randomization [2,27,28,44,22].
A core difficulty here is the challenge of anticipating all the relevant kinds of diversity prior to deployment. The problem of spurious correlations shows that we often fail to include the relevant diversity in D train , even for the relatively simple situation of image classification (Appendix B.2). It seems even more challenging to have diversity in all of the 'fixed' features discussed previously.
Maintaining uncertainty. Goal misgeneralization occurs when we learn an unintended function f θ2 instead of the intended function f θ1 , which both perform similarly on the training data. One solution would be to instead represent all of the functions that behave well on the training data, for example using Bayesian neural networks [23] or ensembling [34], and then defer to humans in situations where these functions disagree with each other [21,37]. This is similar to prior work demonstrating the importance of reward uncertainty in reward learning settings [24,25]. Beyond computational challenges, an important challenge is that such approaches may be too conservative if unanimous agreement is required [37], or heavily dependent on a chosen prior if Bayesian averaging is used.
Understanding and improving inductive biases and generalization. The goal misgeneralization lens does not predict in advance whether training will yield the intended or misgeneralized goal: it only tells us that both possibilities can occur. However, inductive biases will typically make one of these much more likely in practice. Thus, a better understanding of inductive biases and generalization within our models and training algorithms [31,51,5,12] could potentially enable predicting when goal misgeneralization would happen. With a sufficiently good understanding, we may be able to change the inductive biases in order to favor the intended goal.

Techniques targeting deception
One response to the approaches above might be that guaranteeing complete robustness is impossible due to the no-free-lunch theorem [49, Section 5.1]. However, we expect the worst failures of goal misgeneralization to occur when the model actively tricks us into believing that it is doing what we want (as in the case of the deceptive model A2 from Section 4). In these cases, the model 'knows' that its actions are not what we intended (without this knowledge, there would be no reason for the model to deceive us). This allows us to design techniques that target these cases where the model 'knows' its behaviour is bad. Interpretability. Our model (Section 2.2) assumes that the scoring function s depends only on the model outputs. Interpretability techniques [40,9] could give us insight into model computation, such that we could select models that produce good outputs for good reasons [39,26]. Such techniques are particularly helpful in cases where the model's internal computation encodes knowledge that its actions are not what we intended, and would enable penalizing deception in such models. We note that we may then learn a deceptive model that can also deceive the interpretability techniques.
Recursive evaluation. In recursive evaluation approaches, evaluation of models is assisted by other models. Debate [30,4] instantiates this idea with a two-player zero-sum game in which the AI players are rewarded for credibly demonstrating flaws in the other player's actions. Iterated amplification [15] and recursive reward modeling [36] also use models to aid evaluation of other models. These proposals often focus on the case where the original and evaluation models share weights and/or activations. Ideally, if the original model 'knows' that it is behaving badly, this information is also known and can be revealed by other models, which would allow penalizing such behaviour.

Related work
Existing examples of goal misgeneralization. Di Langosco et al. [20] provide empirical demonstrations of goal misgeneralization in three reinforcement learning (RL) environments: CoinRun [16]; Maze; and Keys and Chests. Our work complements theirs in two important ways. First, we provide a definition for goal misgeneralization that applies to arbitrary learning systems, rather than being restricted to RL (Section 2.2). Second, our examples demonstrate goal misgeneralization in qualitatively new settings, including: (a) a setting with a distribution shift induced solely by running the agent longer (Section 3.1), (b) the never-ending RL setting (Section 3.2), (c) large language models without any RL (Section 3.3), and (d) two examples found "in the wild" rather than being hand-designed (Sections 3.4 and 3.5).
Mesa optimization. Hubinger et al. [29] introduce mesa optimization, a type of goal misgeneralization where a learned model implements a search algorithm with an explicitly represented objective. We do not make this assumption -goal misgeneralization can occur without explicit search as well.
Specification gaming. Like goal misgeneralization, specification gaming [33,35] can lead to models that pursue undesired goals. In specification gaming, the feedback provided by s is incorrect in some way (the optimal behavior for this score function is undesirable). Goal misgeneralization arises not because the feedback data was incorrect, but because it was underspecified: there are multiple possible goals that are consistent with the feedback from s on the training data D train . Alignment problems due to misspecification and underspecification are often referred to as 'outer' and 'inner' alignment respectively. Following Ortega et al. [42], we can consider three types of objectives: 1. Ideal objective ('wishes'): The hypothetical objective that describes good behaviour that the designers have in mind.

Design objective ('blueprint'):
The objective that is actually used to build the AI system. This is the designer's attempt to formalize the ideal objective. In the terminology of this paper, this is the scoring function s.

Revealed objective ('behaviour'):
The objective that best describes what actually happens.
In the terminology of this paper, this is the test goal(s) in the test setting and the training goal(s) in the training setting. (Unlike prior work, our definition of goals depends on the environment, and so there is no unique revealed objective.) Problems can arise if there are differences between these objectives. A discrepancy between the ideal and design objectives leads to outer misalignment or specification gaming [33], while a discrepancy between the design and revealed objectives leads to inner misalignment or goal misgeneralization.
Robustness. Goal misgeneralization is a subproblem of the broader robustness problems, whether described as distributional shift [3,Section 7] or underspecification [18] or spurious correlations (Appendix B.2). Robustness problems include any time the model behaves poorly, including cases where the model behaves "randomly". In contrast, in goal misgeneralization, the behaviour must remain coherent. Some of the system's competencies remain intact, allowing it to coherently pursue a misgeneralized goal. This coherence and competence means the behaviour can be higher impact, and hence can be significantly more harmful.

Conclusion
In this paper we have presented a particular subset of robustness failures for learning systems that we call goal misgeneralization. We provided multiple examples of goal misgeneralization failures in deep learning systems, demonstrating that the failure can occur in practice. Overall, we view goal misgeneralization as a plausible mechanism for catastrophic risk from powerful AI systems.
Limitations and future work. While one obvious avenue for future work is to find effective mitigations for goal misgeneralization (Section 5), we would also like to see additional work investigating how likely goal misgeneralization is in practice. Many of our examples were deliberately designed to display goal misgeneralization, and so do not provide much information on this question. We would be excited to see more systematic work: for example, empirically estimating what fraction of "fixed" training features lead to goal misgeneralization when varied at test time, or studying how goal misgeneralization changes with scale.

A An example outside of deep learning: flight booking
While all of our main examples involve deep learning systems, goal misgeneralization can apply to any learning process. In this appendix, we demonstrate with a simple toy example of booking flights, illustrated in Figure 7, in which our AI system learns simply by searching over a space of possibilities until it finds one that achieves high score. Our AI assistant's job is to book flights on the user's behalf. We model the user as optimizing a reward function that is a linear combination of the following features: total cost φ cost , number of stops φ stops , and whether any of the flights happen at night φ night . Thus, our AI simply searches for a setting of weights for features (the learned reward) that predicts Alice's choices well. It then selects future flights by choosing the ones that maximize the learned reward. Given a list of flights x to choose from, it computes argmax flight∈x θ T φ(flight), where θ is the learned reward, and φ is a fixed feature function that extracts the three relevant features of the flight.  Figure 7. This constitutes both the set of goals that are consistent with the training setting, as well as the set of possible θ that could be learned by the AI assistant. Alice's true reward function (intended) is shown in green, and the learned reward function (unintended) in red.
As before, the core issue is that there may be several learned rewards that all score well on the training inputs. For example, the two training inputs visualized in Figure 7 imply the following four constraints: 7θ cost + θ night ≥ 4θ cost + θ stops + θ night 7θ cost + θ night ≥ 8θ cost + θ stops 8θ cost + θ night ≥ 5θ cost + θ stops + θ night 8θ cost + θ night ≥ 7θ cost + θ stops These constraints are satisfied by a large fraction of possible θ, which we visualize in Figure 8.
To define the capabilities of this AI assistant, we need to say what we can get this AI assistant to do with "quick tuning". This is a thorny question, but for the purpose of exposition we will make a simple assumption and say that we can quickly tune the AI system by changing the value of θ directly, since it is a vector of 3 meaningful numbers. Thus, the AI assistant is capable of choosing, say, the most expensive flight (tuning θ to [1, 0, 0]), but it is not capable of choosing the flight with the fewest daytime flights (as this would require a new feature that the AI system does not have).
In the training setting, the behaviour is consistent with the goals corresponding to θ = [−1, −5, −3] and θ = [1, −5, 2], but in the test setting, of these two goals it is only consistent with the one corresponding to θ = [1, −5, 2] 6 . Since the intended goal is the one corresponding to [−1, −5, −3], this is an instance of goal misgeneralization.

B Additional examples of goal misgeneralization
In this appendix we provide additional examples of goal misgeneralization from previous literature.

B.1 Example: CoinRun
We replicate the CoinRun example from Di Langosco et al. [20]. CoinRun [16] is a simple 2-D video game (platformer) where the goal is to collect the coin while dodging enemies and obstacles. By default, the agent spawns at the leftmost end of the level, while the coin is always at the rightmost end. We modify CoinRun to allow the coin to be placed at other locations in the level. This allows us to vary the range of positions where the coin can spawn during both training and testing.
Results are summarized in Figure 9. In line with Di Langosco et al. [20], we find that an agent trained in the default setting has a misgeneralized goal of reaching the end of the level, rather than collecting the coin. When the coin is placed in the middle of the level, the agent reaches the end of the level while typically ignoring the coin, leading to low scores. When we increase the diversity of the training environment by widening the range of coin positions, we find that the agent becomes robust to variations in coin position. For example an agent trained with coins spawning uniformly in the rightmost 20% of the level generalizes at test time to coins spawning anywhere in the level.

B.2 Examples: image classification
The problem of spurious correlations in image classification can be viewed as goal misgeneralization. We give some examples that were taken from the existing literature.  Step 20 the agent needs to avoid the crate and also dodge the green monster. By Step 36 the agent is about to reach the coin. Note that in this episode, the coin is positioned somewhere in the middle of the level.
In a study on automated classification of skin lesions, Narla et al. [38] noted that their algorithm was more likely to classify images with rulers as malignant, instead of focusing on the lesion itself. One can interpret this as goal misgeneralization, in which the test goal is to detect whether a ruler is present. Winkler et al. [55] found a similar issue with patches rather than rulers (see also attempted mitigations in Rieger et al. [47]). In another medical setting, Zech et al. [57] studied generalization of deep learning to classify pneumonia from chest radiographs. They found that the system is more likely to predict pneumonia for images taken at hospitals whose patients are in worse health. One can interpret this as the algorithm having a misgeneralized goal to predict the X-ray generator / hospital, rather than the presence of pneumonia.
In a controversial paper, Wu & Zhang [56] attempt to predict criminality using images of people's faces. Note that we do not endorse such research which risks perpetuating harmful stereotypes [8,7].  [57] Detect pneumonia Detect hospital Image feature detection Criminality [56] Detect criminals Detect smiles Image feature detection Wolves and huskies [46] Detect wolves Detect snow Image feature detection In a critique, Aguera y Arcas et al. [1] suggest that the dataset may contain a bias such that the people in the 'non-criminal' subset are smiling, while those in the 'criminal' subset are frowning. One can interpret this as a misgeneralized goal to predict smiling or frowning.
Ribeiro et al. [46] train a classifier to predict whether an image shows a wolf or a husky, but intentionally only include pictures of wolves that have snow in them. They find the classifier predicts wolf if there is snow, and husky otherwise, regardless of other features. We can interpret this as a misgeneralized goal to detect the presence of snow.

C Experimental details
See sites.google.com/view/goal-misgeneralization for videos of our agents in each of the environments.

C.1 CoinRun
Environment: We modify the CoinRun environment [17] to enable setting the x-position of the coin as a percentage of the level's horizontal length; the y-position is determined by dropping the coin at the given x-position until it lands on something solid. For a coin-position range start of x we uniformly sample x-positions between x% and 100% of the level length in increments of 10%. We use the first 10 8 − 1 procedurally generated levels for training and subsequent levels for evaluation. See Figure 10 for an example rollout. (a) Step 10 Step 11 (c) Step 12 (d) Step 13 (e) Step 14 Figure 11: Example rollout from Monster Gridworld. We render the one-hot encoded observation to RGB, using white for the agent, red for monsters, purple for shields, and green for apples.
Step 10: −1 reward, 0 shields, 5 monsters. The agent is being attacked by monsters, which move towards it.
Step 12: −2 reward, 1 shield, 5 monsters. Monsters move first, and also in this turn they happened to move twice, resulting in -2 reward. The agent then picked up a shield.
Step 13: 0 reward, 0 shields, 4 monsters. The agent used its shield to destroy a monster.
We train agents on 25, 100, and 200 length episodes using 10 seeds and a batch size of 32 episodes for at least 3 billion environment steps each, by which time performance has converged. For each length, we discard the best and worst performing seeds, and evaluate each of the remaining 8 seeds on 100 episodes of length 200. We report mean values for all entities in Figure 2, and show error bands around a 95% confidence interval computed using bootstrapping.
Environment: 10 × 10 gridworld with entities (initially): agent, 10 trees. The agent is provided the full gridworld as a 12 × 12 × 2 one-hot encoded image in its observations. Chopping a tree provides +1 reward. If there are fewer than 10 trees, the probability of a new tree spawning at a random empty location is given by: r = max(r min , r max × log(1 + n current )/ log(1 + n max )) where r min and r max are the minimum and maximum respawn rates, and n current and n max are the current and maximum number of trees. In the experiment shown in Figure 3 we set r min = 10 −6 , r max = 0.3, and n max = 10. In other experiments (not shown), we verify that setting r min = 0 causes complete deforestation from which the agent never recovers.
We train the policy on 256 instances of the environment in the never-ending setting (i.e. with no episodes). Each actor syncs the latest policy every 128 steps.
In Figure 3, we show a single randomly picked run for ease of exposition, but on runs with other seeds we see the same phenomenon. We bin per-step reward into 1000 buckets across all of training, and show error bands around a 95% confidence interval computed using bootstrapping.
We calculate 'affinity' for a given policy π at a given number of trees n by running it in a 200 step episode in an environment with n randomly placed trees, and where trees always respawn immediately.
At the end of the episode, affinity is calculated as: a π n = (ρ π n − ρ b n )/(ρ g n − ρ b n ) where ρ π n is the episodic return from running π, ρ b n is the average episodic return of the random policy with n trees, and ρ g n is the average episodic return of the tree-greedy policy with n trees. Note that the distribution of locations of trees during evaluation may be different from the long-run distribution generated during never-ending training. In particular since trees nearby are easier to chop, the trees left on the grid will tend to be further away from the agent than the random initialisation. Nevertheless, this metric provides a useful proxy for understanding the agent's current policy by comparing its advantage over the random policy to the greedy policy's advantage over the random policy.
In Figure 3b we evaluate each policy 100 times for each n ∈ [1..10], and plot the results as a heatmap.

C.4 Evaluating Linear Expressions
We prompt the Gopher language model (280B parameters) from Rae et al. [45] with ten examples of evaluating linear expressions involving exactly two unknown variables. Variables take values in [−9 . . 9] in all examples and test cases to make the task simpler.
We evaluate this prompted model in Figure 4 by measuring the probability of continuing with a question ('What's ...') or an answer ('The answer is ...'). We use test cases involving zero, one, two, and three unknown variables, and for each number of unknown variables we evaluate 100 randomly generated few-shot prompts. We report mean values with error bars shown around a 95% confidence interval computed using bootstrapping. Table 5 shows an example prompt, and Tables 6a, 6b, 7a, and 7b show randomly sampled dialogue transcripts.

C.5 Cultural Transmission
We use the cultural transmission agent (MEDAL-ADR) from CGI et al. [10]. The agent is paired with programmatically specified bots: expert, anti-expert (always goes to an incorrect next target location), and random. We evaluate on worlds of size 32 × 32m 2 with procedurally generated horizontal and vertical obstacles, and 5 target locations. In Figure 5 we evaluate each pairing with 100 random seeds, and report mean values with error bars shown around a 95% confidence interval computed using bootstrapping. See Figure 1 for example rollouts. Table 5: An example of a few-shot prompt for evaluating linear expressions.