{'fraction_non_alphanumeric': 0.03364444196054323, 'fraction_numerical': 0.015648577656066618, 'mean_word_length': 4.407524932003627, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Atari-5: Distilling the Arcade Learning Environment down to Five Games

The Arcade Learning Environment (ALE) has become an essential benchmark for assessing the performance of reinforcement learning algorithms. However, the computational cost of generating results on the entire 57-game dataset limits ALE's use and makes the reproducibility of many results infeasible. We propose a novel solution to this problem in the form of a principled methodology for selecting small but representative subsets of environments within a benchmark suite. We applied our method to identify a subset of five ALE games, called Atari-5, which produces 57-game median score estimates within 10% of their true values. Extending the subset to 10-games recovers 80% of the variance for log-scores for all games within the 57-game set. We show this level of compression is possible due to a high degree of correlation between many of the games in ALE.


Introduction
The Arcade Learning Environment (ALE) [5] has become the gold standard for evaluating the performance of reinforcement learning (RL) algorithms on complex discrete control tasks. Since its release in 2013, the benchmark has gained thousands of citations and almost all state-of-the-art RL algorithms have featured it in their work [25,16,14,13,12,21]. However, results generated from the full benchmark have typically been limited to a few large research groups. 1 We posit that the cost of producing evaluations on the full dataset is not feasible for many researchers. Additionally, the verification of superiority claims through statistical tests on multiple seeds remains exceedingly rare, likely, due to the high cost.
While technological improvements have reduced computation costs, this decrease has been greatly outpaced by an increase in the scale at which RL algorithms are run. For example, the number of frames used for training to produce state-of-the-art results on ALE increased by more than onethousand times in the five years between 2015's Deep Q-learning model [22] and the more recent Agent57 [2]. Training modern machine learning models can also produce non-trivial amounts of carbon emissions [26], for which we provide some analysis in Appendix A. Previous works have dealt with this challenge by self-selecting ad hoc subsets of games. However, this approach adds bias and makes comparison between works more challenging.
In this paper, 2 we outline a principled approach to selecting subsets of environments from a RL benchmark suite. We show that, when carefully chosen, a surprisingly small subset of ALE can capture most of the useful information of a full run. We present a new benchmark, Atari-5, which produces scores that correlate very closely to median score estimates on the full dataset, but at less than one-tenth the cost. We hope that this new benchmark will allow more researchers to participate in this important field of research, speed up the development of novel algorithms through faster iteration, and make the replication of result in RL more feasible. Our primary contributions are as follows. First, a methodology for selecting representative subsets of multi-environment RL benchmarks according to a target summary score. Second, the introduction of the Atari-5 benchmark. Finally, evidence demonstrating the high degree of correlation between scores for many games within ALE.

Background and Related Work
The Arcade Learning Environment (ALE). Despite deceptively simple graphics, ALE [5] provides a challenging set of environments for RL algorithms. Much of this challenge stems from the fact that ALE contains games spanning multiple genres, including sport, shooter, maze and action games, unlike most other RL benchmarks. This is important because being able to achieve goals in a wide range of environments has often been suggested as a useful definition of machine intelligence [18]. In addition, unlike many other RL benchmarks, the environments within ALE were designed explicitly for human play, rather than machine play. As such, ALE includes games that are extremely challenging for machines but for which we know a learnable (human) solution exists. Human reference scores also provide a means by which 'good' scores can be quantified.
ALE Subsets. Many researchers make use of ALE subsets when presenting results. However, decisions about which games to include have varied from paper to paper. The Deep Q-learning Network (DQN) paper [21] used a seven-game subset, while the Asynchronous Advantage Actor-Critic (A3C) paper [20] used only five of these games. A useful taxonomy was introduced by Bellemare et al. [4], which includes the often-cited hard exploration subset. A more comprehensive list of papers and the subsets they used is given in Appendix B, highlighting that there is currently no standard ALE subset. The key difference in our work is that our selection of games has been chosen by a principled approach to be representative of the dataset as a whole. To our knowledge, this is the first work that investigates the selection and weighting of an RL benchmark suite.
Computationally Feasible Research. As has been pointed out, the computational requirements for generating results on ALE can be excessive [7]. There have been several attempts to reduce this cost, including optimizations to the simulator codebase 3 , and a GPU implementation of the environments [10]. However, as these changes reduce the cost of simulating the environment and not the much higher cost of training a policy, they result in only minor improvements. Asynchronous parallel training [14] partially addresses this by making better use of parallel hardware but still requires access to large computer clusters to be effective. While these improvements have been helpful, they do not go far enough to make ALE tractable for many researchers. Furthermore, our subsetting approach is complementary to the approaches above.
Alternatives to ALE. Since ALE's introduction, many other RL benchmarks have been put forward, such as [8,17,3,23]. However, ALE still dominates research, and at the time of publishing, ALE has more than double the citations of these other benchmarks combined (see Table 1). Minatar [29] addresses some of ALE's issues by creating a new benchmark inspired by Atari that presents agents with objects rather than pixels. This simplification speeds up training but requires previous algorithms Benchmark Citations ALE [5] 2,295 Vizdoom [17] 610 DeepMind Lab [3] 380 Procgen [8] 166 Gym Retro [23] 119 Table 1: Popular discrete-action vision-based benchmarks in RL by citation. Citations are as of Mar 2022.
to be trained on the new environments and lacks the vision part of the task, an important part of ALE's challenge.

Summary Scores on Subsets of Benchmark Suites
When working with a benchmark made up of multiple environments it is often useful to distil the performance of an algorithm, over all environments, down to a single 'summary score' [5]. If this summary score uses scores from only a subset of the environments then the time taken to generate a summary score can be reduced. Ideally this subset summary score would: 1. Preserve order. 2. Provide useful information about the performance of the algorithm. 3. Minimize the number of environments needed to be evaluated.
We formalize these desiderata as follows. Let M = {µ i |i ∈ [1..n]} be an ordered set of n unique environments that make up the benchmark. For some algorithm a we write the empirical score on the i-th environment as a i . Then, for this set of environments, we say that for a pair of algorithms a, b, we have that a ≥ b if and only if a i ≥ b i ∀ i ∈ [1..n] and that a > b if there also exists some i, such that a i > b i . We also define a summary score Φ M on scores from benchmark M as a function Φ M : R n → R reducing a score vector a 1 , a 2 , ..., a n taken from benchmark M , to a scalar indicating performance. For convenience, we write the summary of individual scores for an algorithm, Φ M ( a 1 , a 2 , ..., a n ) as Φ M (a). Finally we define a subset score as a summary score which uses a strict subset of M .

Order Preservation
Ideally a subset summary score would be strictly order preserving, in that if a > b then Φ(a) > Φ(b). Unfortunately this is not possible.
Proposition 1: No subset score can be strictly increasing.
Proof. Let M be the strict subset of environments used by Φ M , then select z ∈ [1..n] where µ z ∈ M \ M . Now consider two algorithms a, b, where a i = b i ∀ i = z, and a z > b z . Hence we have that a > b, but not Φ(a) > Φ(b). Therefore, the next best thing we can do is to require that our subset score be at least weakly order preserving, that is, if a > b we can Proposition 2: Any summary score of the form Proof. See Appendix C.
We therefore find that linear combination of scores, if constrained to non-negative weights, is weakly order preserving. We note that this is not the case for more sophisticated non-linear models, such as deep neural networks, which might reverse the order of algorithms. We also note that any monotonically increasing function φ : R → R can be applied to the individual game scores, which we will take advantage of by using a log transform. We can therefore guarantee that if algorithm a is strictly better than algorithm b, it can be no worse when measured using a weighted subset score, which is, as shown, the best we can do.

Provide Useful Information
As shown, a weighted subset produces a monotonic measure of an algorithm's performance. However, the question of which subset to use and how to weigh the scores remains. A trivial solution would be to weigh each score by 0, which would be fast to generate and weakly order-preserving but completely uninformative. Therefore, we propose that individual scores should be weighted such that the summary score provides meaningful information about the performance of the algorithm. For established benchmarks, meaningful summary performance measures have often already been established. For example, with ALE, the most common summary score is median score performance. Therefore, we suggest that the prediction of an established summary score (target score) provides a sensible strategy for selecting and weighting the environments within the suite. That is, to find the subset of environments that best predicts the target score and weigh them according to their linear regression coefficients.

Procedure
We therefore propose that using linear regression models to predicate a meaningful target summary score provides a good solution to the desiderta outlined above, and formalize the procedure. Given m algorithms a k : k = 1...m together with their individual evaluations s k i ∈ R on environments i ∈ {1 : n} and total/aggregate score t k ∈ R, e.g. t k = s k 1 + ... + s k d or the average or the median. The "training" data set is hence D = {(s k 1 , ..., s k n ; t k ) : k = 1...m}. Let I ⊆ {1 : n} be a subset of games and s I := (s i : i ∈ I) be a corresponding score sub-vector. We want to find a mapping f * I : R I → R that best predicts t from s I , i.e. formally f * I = arg min f m k=1 (t k − f (s k I )) 2 . Since for I = {1 : d}, the best f is perfect and linear, it is natural to consider linear f , which leads to an efficiently solvable linear regression problem. We further want to find small I with small error. We hence further minimize over all I ⊆ {1 : n} of some fixed size |I| = C. The optimal set of games I * with best linear evaluation function f * based on games in I * hence are

Application to ALE
This section provides details for the application of our method to the popular ALE benchmark, but note that this method could also be applied to other benchmark suites such as the Procgen [8], MuJoCo [27], or even benchmarks outside of RL.

Data and Processing
We used the website paperswithcode 4 as the primary source of data for our experiments. This website contains scores for algorithms with published results on various benchmarks, including ALE. The dataset was then supplemented with additional results from papers not included on the website. Additions included various algorithms, such as evolutionary algorithms and shallow neural networks. A full list of algorithms and their sources contained within the dataset can be found in Appendix D.
We removed any algorithms that had results for fewer than 40 of the 57 games in the standard ALE dataset on the grounds that a reasonable median estimate could not be obtained. This reduced our dataset of 116 algorithms down to 62. When fitting models for subsets, any algorithm missing a score for a required game was excluded for that subset. We also excluded any games with results for fewer than 40 of the remaining 62 algorithms. The only game meeting this criteria was Surround. All scores were normalized using the standard approach from [5] where r i and h i are the human and random scores for environment i respectively, which we include in Appendix 12. We found that, even after normalization, scores differed across games by many orders of magnitude and produced non-normal residuals. We, therefore, applied the following transform to the normalized scores, producing log normalised scores. Clipping was required as algorithms occasionally produce scores slightly worse than random, generating a negative normalized score. Both input and target variables were transformed using φ, and all loss measures were calculated in transformed space. This transformation also has the effect that larger scoring games would not dominate the loss.

Predictive Ability of Individual Games
To better understand the games within ALE, we produced single-game linear models where the score from a single environment was used to predict the median Atari-57 score. We then ranked games according to their predictive ability as measured by their R 2 . Due to a small parameter count (n=2), cross-validation was not used.

Subset Search
To assess the best selection of games for each subset, we evaluated all subsets of the ALE games of size five three or one that met our criteria as defined in Section 4.1. 5 Subsets were evaluated by fitting linear regression models to the data using the log normalized scores of games within the subset to predict the median overall, selecting the model with the lowest 10-fold cross-validated mean-squared-error. For these models, we disabled intercepts as we wanted a random policy to produce a score of 0.
To make the best use of the 57 games available, we carefully ordered the subset searches to maximize the performance of the most common subset size while maintaining the property that smaller subsets are true subsets of their larger counterparts. All subsets of five games were evaluated, with the best subset selected as Atari-5. The Atari-3 subset was then selected as the best triple game subset of the Atari-5 subset and Atari-1 as the best single-game subset of Atari-3. Doing this ensures that only five games are included in the test set and also that evaluations on Atari-1 or Atari-3 can more easily be upgraded to Atari-5 if required. For the validation subsets, we selected the best three games, excluding the games used in Atari-5, as the validation set. We then extended this set to 5 games by choosing the best two additional games. Finally, the ten-game Atari-10 set was generated by selecting the best five additional games, not in Atari-5 nor Atari-5-Val.
Searching over all subsets took approximately 1-hour on a 12-core machine. No GPU resources were used. Source code to reproduce the results are provided in the supplementary material.

Prediction over All Game Scores
To better understand how much information our subsets capture about the 57 game dataset, we trained 57 linear regression models for the best five-game and ten-game subsets, predicting log-normalized scores for each game within the 57-game set. We then evaluated each of the 57 models according to their R 2 , allowing us to identify well or poorly captured environments by the subset. These models also enable score predictions for any game within the 57-game set.

Results
In this section, we present the results of our subset search on the ALE dataset using the Atari-57 median score as the target metric.  Table 2: Performance, and included games, for each of the best models for each subset size.

Single Environment Performance
We found a significant difference in the ability of games within the ALE benchmark suite to predict median score estimates ( Figure 2). For single-game subsets, Zaxxon performed best with R 2 = 0.903, and Surround the worst with R 2 = 0.002. The environment Name this Game was found to have the interesting property that its intercept was close to 0 (-0.03) and its scale close to 1 (1.01). As a result, normalized scores from Name This Game provide relatively good estimates for the median performance of an algorithm, by themselves, without modification.
We also note that several games have close to no predictive power when used by themselves. In the case of Pong, Surround, and Tennis, this is most likely due to a low score cap, which most algorithms easily achieve. Games within the hard exploration subset also performed poorly at predicting the median score due to many algorithms achieving no better than random performance.

Optimal Subsets
As a baseline reference, we include results for the 7-game set used in the DQN paper [22]. The result demonstrates the importance of game selection in that our single-game model outperforms the 7-game baseline set used by the DQN paper. Table 2 gives the results for the top-performing regression model for each subset size. Coefficients are provided in Appendix E.
We also approximated the expected relative error of the median score, using log e (10) times the mean absolute error in log space, which, for small residuals, is a good approximation of the relative error. 6 We found that Atari-5 is generally within 10% of the true median score. In Table 3, we give the performance of popular algorithms on the Atari-5 benchmark. 6 The authors were unable to find a reference for this, and so have provided a proof in Appendix G.
S u rr ou n d S ol ar is Fr ee w ay D ou b le D u n k B ox in g Po n g V en tu re Pr iv at e E ye Te n n is Pi tf al l E n d u ro R ob ot an k B ow lin g K an g ar oo S ki in g B re ak ou t M on te zu m a R ev en g e A tl an ti s Tu ta n kh am V id eo Pi n b al l G ra vi ta r C ra zy C lim b er Fi sh in g D er b y S ta r G u n n er  Table 3: Results for MuZero [25], Agent57 [2], Ape-X [14], IQN [9], and Rainbow DQN [13]. Atari-5 predictions fall close the true Atari-57 median scores, and the ranking of the algorithms is generally preserved.

Robustness to Environmental Settings
To assess the impact of significantly different environmental settings, we used Atari-5 to evaluate the algorithm Go-Explore [11], which was not included in our training set. Go-Explore uses 400,000 frame time limits for each game, rather than the 108,000 used by the algorithms in our dataset. For many games, this difference fundamentally changes the skills the game requires and, importantly, the maximum possible score. For example, in the Space Invaders-like game Demon Attack, a shorter time limit requires an agent to take risks to score points more quickly. In contrast, an extended time limit emphasises staying alive instead. Given the differences, we were surprised to find that Atari-5 produced an overestimate of only 12.2% (an estimate of 1,620 compared to 1,446). While this result supports the robustness of Atari-5 to provide useful performance information, even when environmental settings have been changed, we still recommend that it not be used in these cases.

Fairness of Atari-5 Scores
To assess the fairness of Atari-5, we split the algorithms with scores for all five games in the Atari-5 dataset into tertiles, ordered by their true median score performance. We looked for differences in accuracy between the groups by comparing the average absolute relative error and looked for bias by comparing the average relative error. No statistically significant difference was found between any pairs using the standard p=0.05 threshold and Wilson's two-sided T-test.  Figure 3: Fairness of the dataset. No statistically significant differences were found in either the accuracy, or the bias of the estimates across the low, medium and high performing algorithms.

Predicting All Game Scores
We found that Atari-5 was able to produce surprisingly accurate score estimates for many of the other games in the ALE benchmark, as shown in Figure 4. There were 17 games which showed a high degree of correlation, with R 2 greater than 0.8, and together, the models explain 71.5% of the variance in log-scores within the dataset.
Inspired by this result, we tested the 'vital few, trivial many' principle [15], 7 by repeating the experiment with the Atari-10 dataset (17.5% of the environments) and found that those games could explain 80.0% of the variance of the full 57-game set. Because of this, in many cases, other performance measures (for example, the number of games with human-level performance) could potentially be approximated using the Atari-10 game set. Model parameters for predicting scores for these games is given in Appendix E. We also note that while the Atari-5 games predict many of the games well, they perform poorly in Bellemare's hard exploration games [4].
S o la ri s V e n tu re Fr e e w a y T e n n is K a n g a ro o S ki in g P o n g P it fa ll  Figure 4: The five games selected in Atari-5 are able to predict accurate scores for many of the other games within the ALE benchmark. However, hard exploration exploration games are less well predicted by the subset.

Structure Within in the ALE
While the ALE dataset contains a diverse range of games, there are still some games within the dataset that are highly correlated. Therefore, we assessed the similarity between scores in games in the ALE benchmark by calculating the Pearson Correlation Coefficient (PCC) [6] on the log-transformed normalized game scores for each game pair in the 57-game dataset. We considered a pair with a coefficient above 0.9 to be highly correlated. The most correlated twenty-four pairings are shown in Figure 5, with highly correlated pairs depicted with bold edges. We found that the two most correlated games were Alien and Ms. Pacman, which was not surprising as these games are both Pacman-like games with very similar rules and primarily differ only in their graphical presentation. We also separated games into categories based on their genre 8 and note that games of similar categories are often clustered together.
We also checked to see if any games were negatively correlated and found Skiing scores were moderately negatively correlated with Alien (-0.51), Frostbite (-0.55), Ms. Pacman (-0.51), and River Raid (-0.50). Skiing is unlike most other ALE games in that all score is deferred until the terminal state of the episode and that the locally optimal but globally sub-optimal policy of racing to end ignoring penalty flags is easily learnable. This suggests that Skiing may require novel approaches to solve.

Case Study
Our experimental results show strong results for Atari-5 on the dataset used to fit the model. In addition, K-fold cross-validation scores indicate that our model is not overfitting the training data and suggest the model should generalize to unseen data well. However, changes in algorithms or environmental settings may result in poorer performance when applied to unseen data. For this reason, we performed a case study where we applied Atari-5 retroactively to a recent paper not included in our training set. We then asked the following question. If this paper had used Atari-5 instead of the full ALE, would the results have changed materially?
We set up our experiment as follows. First, we searched for accepted papers submitted to AAAI-22 that referenced ALE. We then selected the first paper,which contained results for no less than 40 games and which included results for the five games in the Atari-5 benchmark. The first paper we found meeting this requirement was Conjugated Discrete Distributions for Distributional Reinforcement Learning [19], which contains results for C51, IQN, and Rainbow, as well as their own algorithm C2D. 9 We defined two evaluation metrics, which are relative error on predicted median score as compared to the true median score, and the inversion count between the order of the algorithms under the median score, as compared to the order under Atari-5. We used the arXiv preprint version of the paper, which included the supplementary material containing the required results.
The results of our study are given in Table 4. Relative error on these four algorithms was higher than expected (12.6%-26.0%, average of 18.9%). However, we found that Atari-5 scores consistently underpredicted the true results by a similar ratio (≈ 0.80). We hypothesise that this may be due to differences in the environmental settings (evidenced by Rainbow scoring 147 on their experiments compared to 223 in our dataset). Since it is often useful to measure the performance of an algorithm relative to some baseline, we also considered the scores normalised to the best performing algorithm, Rainbow. We present the results in and Table 5. Because Atari-5 is consistent in its under prediction, with this adjustment, relative error for all models fell below 10% (3.5% -9.0%, average of 6.7%). We note only one inversion in the order: the swapping of C51, and IQN.  Table 5: Median and Atari-5 scores normalized to the Rainbow result. These results indicate the relative performance of each algorithm as compared to Rainbow.
Given these results, we conclude that, had Atari-5 been used instead of the full ALE dataset, while the median score would have varied substantially (18.9%), the relative performance differences would have been similar (6.7%). This degree of error was not significant enough to change the paper's outcome, which was that C2D outperformed both C51 and IQN and underperformed Rainbow. We, therefore, conclude that had Atari-5 been used, the results would not have changed materially.

Discussion
In this section, we outline the recommended use of the Atari-5 benchmark, discuss its primary value and limitations, and consider the broader impact of our work. We intend Atari-3-Val to be used for hyperparameter tuning and algorithm development and Atari-5 to be used to generate evaluation results. In some cases, researchers might want to include results for specific games that demonstrate a strength or weakness of their algorithm while also providing a measure of general performance in the form of Atari-5. Because Atari-5 and median scores differ in their bias, it is recommended to always compare Atari-5 scores with Atari-5 scores, and not against median scores directly. For this reason, we have provided Atari-5 scores for common algorithms in Table 3.
Atari-5 provides three key points of value. First, it captures a significant portion of the information gained from a full ALE evaluation using one-tenth of the evaluations. Second, it provides a standard subset to ALE, selected using a principled process. Third, it provides backwards compatibility with prior work allowing results on previous algorithms to be generated after the fact, so long as scores for the five games were included.

Limitations
Algorithms included in our training dataset are very diverse and include neural network and nonneural network approaches. Because of this, Atari-5 generates results robust to algorithmic and training choices, such as how many frames an algorithm was trained for. However, any changes made to the environment, importantly the maximum number of frames, could affect the result, as they modify the optimal policy and potentially the maximum score. We also note that stochasticity in the form of the probability of repeating actions is an environmental change but that our dataset contains both stochastic and deterministic variants. This does not seem to make a significant difference to the prediction accuracy.
We also note that Atari-5 does not do well at incorporating hard-exploration games (see Figure  4). We, therefore, recommend, for algorithms where this matters, to run individual tests for these games. Researchers could then establish their algorithm's general performance via Atari-5, while also demonstrating a specific ability on hard-exploration problems by giving individual results on those games.

Broader Impact
We believe that Atari-5 plays a vital role in increasing access and diversity in RL research, thus increasing reproducibility. Our work aims to address several issues that currently affect RL researchers. One of the barriers to working in RL research is the high level of computation required to produce results on common benchmarks. Researchers who do not have access to tens of thousands of dollars of compute capacity cannot evaluate their algorithms on established benchmarks, inhibiting their ability to publish their work. This has led to a lack of diversity in the literature, evidenced by the fact that only two companies have been able to produce results on the full dataset in six years. We believe Atari-5 is an important step in the right direction, as it brings Atari games inside the sphere of computationally accessible benchmarks.

Conclusions
We have introduced a new benchmark, Atari-5, that provides median score estimates to within 10% of their true values but at less than one-tenth the cost. A key feature of Atari-5 is that, as a subset of ALE, it can be applied retroactively to prior results, without retraining, so long as scores for all five games are provided. We have also shown that our extended benchmark Atari-10 can generate useful score estimates for all games within the ALE dataset. We hope this new benchmark can offset the trend of increasingly high computation requirements required to develop and reproduce results on the ALE benchmark and increase RL participation while providing backwards compatible results to prior work.

A Estimation of Cost to Reproduce for Selected Papers
Retroactive estimation of costs and carbon emissions can be difficult [24]. Due to missing details in the publications, our estimations required many assumptions about the hardware and time needed to reproduce a result. Our estimates do not include overheads such as air-conditioning, testing, hyper-parameter searches, or ablation studies. Therefore these estimates should be taken as useful lower bound estimates on the cost to reproduce a work. We also note the large difference between preemptive and standard costs which are approximately one-third of the cost. However, preemptive computing may not always be feasible due to congestion near conference deadlines. We took the computing requirements from the papers and used Google Cloud Compute to estimate the cost of reproducing the work. 10 For some older algorithms, it may be possible to reduce the costs given here by training on more modern hardware (for example, an A100 or TPU). However, if the algorithm is CPU constrained, this could increase, not decrease, the cost due to the underutilization of the device. Therefore we estimate reproduction costs with our best guess at the hardware that would have been used at the time of publication. A summary of the calculations and results is given in Table 6. In addition, we detail the assumptions made for each algorithm in the paragraphs below. Rainbow DQN Rainbow DQN required 10 days on a single GPU for each of the 57 environments. [13] The hardware used is not mentioned in the paper so we assume a V100, with an E2-standard-4 (16GB) instance, 11 which has a GPU/CPU cost per hour of $2.48/$0.134 (standard), and $0.74/$0.04 (preemptive). We also note that [7] also produced estimates for Rainbow DQN, but used 5-days of GPU time instead of 10.
Agent-57 Agent-57 trains at ≈ 260 environment steps per second per actor with 256 actors [2]. To generate the 90B frames needed would then take 375 hours. The paper states that a single GPU is used, and for the purposes of our estimate, we assume that this also requires 375 hours. 12 The hardware used is not stated, so we assume an A100, which is the best available GPU at the time of their publication. We assume a 64 CPU machine (N2-highcpu-64) would be needed as our observation is that Atari environments typically run at 1,000 environment steps per CPU. The GPU/CPU cost per hour is $2.9333/$2.2294 (standard) and $0.88/$0.555 (preemptive).

Power Usage and Carbon Emissions
We estimate the power required to reproduce the Agent-57 result as follows. A single A100 uses up to 400W. 13 We use 215W as the estimated power consumption for the CPU, which is taken from the Xeon Phi 7210 specifications. 14 We do not include estimates for air-conditioning, power supply inefficiency, or other components such as RAM, networking, and storage. The use of TDP is only an approximate measure as devices frequently run under the TDP limit and may also occasionally run over it too. Using the estimates above, the total power required to reproduce a result would be 13.166mWh. Because our compute estimations were based on Google Cloud Compute servers in the U.S., we used the 2019 U.S. carbon emissions rate of 0.92 pounds per kWh. 15 We note that this could be higher in developing countries. This equates to approximately 6.05 tones of CO2 per 57-game run.

C Proof of Proposition 2
We prove the more general case of Proposition 2 using some fixed (weakly) increasing φ : R → R, where Proposition 2 can be recovered by selecting φ(x) = x.
Consider two algorithms a, b, and a set of scores for each of n environments a 1 ...a n , b 1 ...b n , where a i is the score on the i-th environment for algorithm a, and b i is the score on the i-th environment for algorithm b. Given some (weakly) increasing function φ : R → R, and constants c i ∈ R ≥ 0, we have that Since this holds for all i ∈ [1.
.n] we have as required.

D Algorithms Included in the Dataset
We included in our dataset only algorithms with results on 40 or more ALE games. They are listed in Table 8, along with how many games each algorithm had results for. Games that are not part of the 57-game canonical set were not included. The names are provided from PapersWithCode, and some algorithms appear multiple times as they included multiple configurations.

E Models for Individual Games
We provide coefficients for each of the Atari-X and Atari-Val-X models in Table 9, as well as models for predicting scores for all games within the 57-game ALE dataset (Table 10, 11). An estimated normalized game score (s est ) can be calculated using the following formula s est = 10 slog − 1 where s i are the normalized scores for the appropriate game using normalization constants taken from Table 12. Not all models make use of the intercept constant c.

F Normalization Constants
For completeness we include the average human reference scores used for normalization in Table 12 which have been taken from [2]. Note, these differ from the professional human scores given by [22].

G Log Residuals as an Approximation to Relative Error
Here we show that log residuals are approximations to the relative error. For a residual using a biased log transform given by where y is the true value, andŷ the prediction, with y,ŷ ∈ R ≥ 0 and ∈ R > 0 we have δ := log( +ŷ) − log( + y) = log( +ŷ + y ) = log(1 + ( +ŷ + y − 1)) ≈ +ŷ + y − 1 (for +ŷ + y close to 1) which, for y, approximates the relative error. When using logarithms other than the natural logarithm, the log residual δ must first be multiplied by the natural log of the base used.