{'fraction_non_alphanumeric': 0.033682509611634256, 'fraction_numerical': 0.015016437287568954, 'mean_word_length': 4.99549022882913, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 7, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark

Commonsense AI has long been seen as a near impossible goal -- until recently. Now, research interest has sharply increased with an influx of new benchmarks and models. We propose two new ways to evaluate commonsense models, emphasizing their generality on new tasks and building on diverse, recently introduced benchmarks. First, we propose a new multitask benchmark, RAINBOW, to promote research on commonsense models that generalize well over multiple tasks and datasets. Second, we propose a novel evaluation, the cost equivalent curve, that sheds new insight on how the choice of source datasets, pretrained language models, and transfer learning methods impacts performance and data efficiency. We perform extensive experiments -- over 200 experiments encompassing 4800 models -- and report multiple valuable and sometimes surprising findings, e.g., that transfer almost always leads to better or equivalent performance if following a particular recipe, that QA-based commonsense datasets transfer well with each other, while commonsense knowledge graphs do not, and that perhaps counter-intuitively, larger models benefit more from transfer than smaller ones. Last but not least, we introduce a new universal commonsense reasoning model, UNICORN, that establishes new state-of-the-art performance across 8 popular commonsense benchmarks, aNLI (87.3%), CosmosQA (91.8%), HellaSWAG (93.9%), PIQA (90.1%), SocialIQa (83.2%), WinoGrande (86.6%), CycIC (94.0%) and CommonsenseQA (79.3%).


Introduction
In AI's early years, researchers sought to build machines with common sense (McCarthy 1959); however, in the following decades, common sense came to be viewed as a near impossible goal. It is only recently that we see a sudden increase in research interest toward commonsense AI, with an influx of new benchmarks and models (Mostafazadeh et al. 2016;Talmor et al. 2019;Sakaguchi et al. 2020).
This renewed interest in common sense is ironically encouraged by both the great empirical strengths and limitations of large-scale pretrained neural language models. On one hand, pretrained models have led to remarkable progress across the board, often surpassing human performance on Copyright © 2021, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. Figure 1: Cost equivalent curves comparing transfer learning from GLUE, SUPERGLUE, and RAINBOW onto COM-MONSENSEQA. Each curve plots how much training data the single-task baseline (the x-axis) needs compared to the multitask method (the y-axis) to achieve the same performance (shown on the top axis in accuracy). Curves below the diagonal line (y = x) indicate that the multitask method needs less training data from the target dataset than the single-task baseline for the same performance. Thus, lower curves mean more successful transfer learning.
leaderboards (Radford et al. 2018;Devlin et al. 2019;Liu et al. 2019b;Raffel et al. 2019). On the other hand, pretrained language models continue to make surprisingly silly and nonsensical mistakes, even the recently introduced GPT-3. 1 This motivates new, relatively under-explored research avenues in commonsense knowledge and reasoning.
In pursuing commonsense AI, we can learn a great deal from mainstream NLP research. In particular, the introduction of multitask benchmarks such as GLUE (Wang et al. 2019b) and SUPERGLUE (Wang et al. 2019a) has encouraged fundamental advances in the NLP community, accelerating research into models that robustly solve many tasks and datasets instead of overfitting to one in particular. In contrast, commonsense benchmarks and models are relatively nascent, thus there has been no organized effort, to date, at administering a collection of diverse commonsense benchmarks and investigating transfer learning across them.
We address exactly this need, proposing two new ways to evaluate commonsense models with a distinct emphasis on their generality across tasks and domains. First, we propose a new multi-task benchmark, RAINBOW, to facilitate research into commonsense models that generalize well over multiple different tasks and datasets. Second, we propose a novel evaluation, the cost equivalent curve, that sheds new insight on how different choices of source datasets, pretrained language models, and transfer learning methods affect performance and data efficiency in the target dataset.
The primary motivation for cost equivalent curves is data efficiency. The necessary condition for state-of-the-art neural models to maintain top performance on any dataset is a sufficiently large amount of training data for fine-tuning. Importantly, building a dataset for a new task or a domain is an expensive feat, easily costing tens of thousands of dollars (Zellers et al. 2018). Therefore, we want the models to generalize systematically across multiple datasets, instead of relying solely on the target dataset.
Shown in Figure 1, the cost equivalent curve aims to answer the following intuitive question: how much data does a transfer learning approach save over the baseline that doesn't benefit from transfer learning? We provide a more detailed walk-through of this chart in §2. As will be seen, cost equivalent curves have distinct advantages over simple evaluations at the full dataset size or classical learning curves drawn for each method and dataset separately, as they provide more accurate comparative insights into data efficiency in the context of multitasking and transfer learning.
We leverage these new tools to reevaluate common approaches for intermediate-task transfer (Pruksachatkun et al. 2020). Through extensive experiments, we identify multiple valuable and sometimes surprising findings, e.g., that intermediate-task transfer can always lead to better or equivalent performance if following a particular recipe, that QA-based commonsense datasets transfer well to each other, while commonsense knowledge graphs do not, and that perhaps counter-intuitively, larger models benefit much more from transfer learning compared to smaller ones.
In addition to the empirical insights, we also introduce a new universal commonsense reasoning model: UNICORN, establishing new state-of-the-art performances across 8 benchmarks: αNLI (87.3%) (Bhagavatula et al. For reproducibility, we publicly release the UNICORN model and code, all the experimental results, and the RAIN-BOW leaderboard at https://github.com/allenai/rainbow. 2 The CYCIC dataset and leaderboard are available at https:// leaderboard.allenai.org/cycic.

Cost Equivalent Curves
Cost equivalent curves show equivalent costs between the single-task baseline and a new transfer-based approach. In this work, we define cost as the number of training examples in the target dataset. Intuitively, we want to measure how many examples the new approach needs to match the singletask baseline's performance as the amount of data varies. Figure 1 illustrates cost equivalent curves with COMMON-SENSEQA as the target dataset. The x-axis shows the number of examples used by the single-task baseline, while the y-axis shows the examples from the target dataset used by the new multitask method. The curve is where they achieve the same performance. The numbers on top of the figure show the performance corresponding to the number of baseline examples from the x-axis. For example, with 4.9k examples, the baseline achieves 70% accuracy. For any number of examples the baseline might use, we can see how many examples the new approach would require to match it. In Figure 1, to match the baseline's performance on ∼10k examples, multitasking with RAINBOW requires about 5k, while multitasking with GLUE requires more than 10k. Thus, lower is better, with curves below the diagonal (y = x) indicating that the new method improves over the baseline.
The construction of cost equivalent curves makes one technical assumption: the relationship between performance and cost is continuous and strictly monotonic (i.e., increasing or decreasing). This assumption holds empirically for parameters, compute, and data (Kaplan et al. 2020). Thus, we can safely estimate each learning curve with isotonic regression ( Barlow et al. 1972), then construct the cost equivalent curve by mapping each dataset size to the baseline performance, finding the matching performance on the new method's curve, and seeing how many examples are required.
Cost equivalent curves visualize how a new approach impacts the cost-benefit trade-off, i.e. examples required for a given performance. This reframes the goal from pushing up performance on a fixed-size benchmark to most efficiently solving the problem. While we focus on data efficiency in this work, the idea of cost equivalent curves can be applied to other definitions of cost as well (e.g., GPU compute).

RAINBOW
We define RAINBOW, a suite of commonsense benchmarks, with the following datasets. To keep evaluation clean-cut, we only chose multiple-choice question-answering datasets.

Empirical Insights
We present results from our large-scale empirical study, using pretrained T5-LARGE to transfer between datasets. We've grouped our findings and their relevant figures around the four following thematic questions.

What's the Best Approach for Transfer?
We compare three recipes for intermediate-task transfer: (1) multitask training (Caruana 1995): training on multiple datasets (including the target dataset) all at once,  methods perform similarly; however, on the rest, sequential and multitask fine-tune training greatly improve data efficiency. While sequential and multitask fine-tune training are often comparable, sequential training appears to be slightly more data efficient, both from comparing cost equivalent curves in Figure 2 and full dataset performance in Table 1.
Finding 2: Sequential training rarely hurts performance. While multitask training doesn't always beat the single task baseline, sequential and multitask fine-tune training uniformly outperform it-for all RAINBOW tasks and dataset sizes (including full datasets). This pattern mostly holds with other source and target tasks, especially for sequential training which rarely significantly harms performance.
Finding 3: Multitask training helps most often in the lowdata regime. One mystery researchers currently face is the inconsistent effect of multitask learning: sometimes it helps, sometimes it hurts, sometimes it has no effect. Cost equivalent curves reveal one potential explanation: multitask learning tends to help when data is scarce, but may hurt performance if data is plentiful. In Figure 2, all cost equivalent curves initially require fewer examples than the single-task baseline (the y = x line), while on some tasks (HELLASWAG and WINOGRANDE) multitasking eventually needs more data than the baseline.  Figure 3 presents cost equivalent curves and Table 2 provides full dataset numbers.
Finding 4: RAINBOW transfers best for common sense. Across all six RAINBOW tasks and all training set sizes, the RAINBOW tasks transfer better to each other than GLUE and SUPERGLUE do to them. The same result also holds for the popular benchmark COMMONSENSEQA when multitask training ( Figure 1); though, when multitasking with JOCI (Zhang et al. 2017), an ordinal commonsense variant of natural language inference, RAINBOW appears either not to help or to slightly hurt data efficiency-potentially more so than GLUE and SUPERGLUE. 4 Finding 5: Only RAINBOW uniformly beats the baseline.
With sequential training and T5-BASE or larger, RAINBOW improves data efficiency and performance for every task considered. Importantly, this pattern breaks down when multitask training, for which no multiset uniformly improved performance.

Does Model Size Affect Transfer?
Most of our exhaustive experiments use T5-LARGE (770M parameters), but in practice, we might prefer to use smaller models due to computational limitations. Thus, we investigate the impact of model size on intermediate-task transfer using the T5-BASE (220M parameters) and T5-SMALL (60M parameters) models. Figure 4 presents the results for transferring with different model sizes from RAINBOW to COMMONSENSEQA.  Table 3: A comparison of dev accuracy when generatively training on knowledge graphs in a multitask setup using T5-LARGE.
Finding 7: Sequential training wins across model sizes. Figure 4 expands Finding 1, that sequential training generally matches or beats the other transfer approaches, by supporting it across model sizes. In all three plots, sequential training appears in line with or better than the other transfer methods.

Can Models Transfer from Knowledge Graphs to QA Datasets?
Due to reporting bias (Gordon and Van Durme 2013), common sense rarely appears explicitly in text, though it does appear implicitly. While language models learn much of the common sense implicit in natural language (Trinh and Le 2018), crowdsourced and expert curated knowledge might provide complementary information. To investigate, we used two popular common sense knowledge graphs, CONCEPT-NET (Speer, Chin, and Havasi 2017) and ATOMIC (Sap et al. 2019a), to create additional knowledge graph generation tasks (Bosselut et al. 2019). In the forward direction, the model predicts the object given the subject and relation concatenated in XML tags. In the backward direction, the model predicts the subject given the object and relation. The results are summarized in Figure 5 and Table 3.
Finding 8: Knowledge graph multitasking shows little impact. The results are generally negative. Only SO-CIALIQA benefits, which might come from the use of ATOMIC during its construction. We offer two possible explanations: the serialized language from the knowledge graphs is not in a QA format, and the knowledge graph completion task is generative while all other tasks are discriminative. These discrepancies may present too large an obstacle for effective transfer. Our findings encourage future research to better close the gap between knowledge graphs and datasets. Given sequential training's strength, as exemplified in Findings 1, 2, and 7, it may lead to different results than the multitask transfer we explore here.
Finally, we present our universal commonsense reasoning model, UNICORN. Motivated by Finding 1, our primary goal with UNICORN is to provide a pretrained commonsense reasoning model ready to be fine-tuned on other downstream commonsense tasks. This is analogous to how off-the-shelf T5 models are multitasked on NLP benchmarks such as GLUE and SUPERGLUE as part of their pretraining. In order to see the limit of the best performance achievable, we start by multitasking T5-11B on RAINBOW. We then trained UNICORN on each task individually, except for WINOGRANDE which required separate handling since it evaluates models via a learning curve. For WINOGRANDE, we multitasked the other five RAINBOW datasets and then trained on WINOGRANDE. 5 In each case, we used the same hyper-parameters as UNICORN did during its initial multitask training, extending each of the 8 combinations tried at that stage. The best checkpoints were chosen using accuracy on dev.

Related Work
Scaling Laws In contemporary machine learning, simple methods that scale often outperform complex ones (Sutton 2019). Accordingly, recent years have seen a sharp rise in compute used by state-of-the-art methods (Amodei and Hernandez 2018). Performance gains from increasing data, parameters, and training are not only reliable, but empirically predictable (Hestness et al. 2017;Sun et al. 2017;Rosenfeld et al. 2020;Kaplan et al. 2020). For example, Sun et al. (2017) found that models need exponential data for improvements in accuracy. 7 These observations, that scaling is reliable, predictable, and critical to the current successes, motivate our focus on evaluation based on cost-benefit trade-offs, i.e. the cost equivalent curve.
Commonsense Benchmarks Rapid progress in modeling has led to a major challenge for NLP: the creation of suitable benchmarks. Neural models often cue off statistical biases and annotation artifacts to solve datasets without un-5 While sequential training for the RAINBOW tasks would likely yield the best results, it would have required much more compute. 6 All tasks use accuracy for evaluation except WINOGRANDE which uses area under the dataset size-accuracy learning curve. 7 Eventually, models saturate and need super-exponential data.

Conclusion
Motivated by the fact that increased scale reliably improves performance for neural networks, we reevaluated existing techniques based on their data efficiency. To enable such comparisons, we introduced a new evaluation, the cost equivalent curve, which improves over traditional learning curves by facilitating comparisons across otherwise hardto-compare contexts. Our large-scale empirical study analyzed state-of-the-art techniques for transfer on pretrained language models, focusing on learning general, commonsense knowledge and evaluating on common sense tasks. In particular, we introduced a new collection of common sense datasets, RAINBOW, and using the lessons from our empirical study trained a new model, UNICORN, improving state-of-the-art results across 8 benchmarks. We hope others find our empirical study, new evaluation, RAINBOW, and UNICORN useful in their future work. Improving language understanding by generative pre-training.

A Cost Equivalent Curves
Section 2 discusses the intuitions, assumptions, and visualization of cost equivalent curves at a high level. This appendix provides additional discussion as well as technical details for implementing cost equivalent curves.
The aim of cost equivalent curves is to visualize how an innovation impacts a cost-benefit trade-off, in a compact and intuitive way. Since cost equivalent curves are more general than the use case explored in this work (dataset size / performance trade-offs), we'll introduce more general terminology for discussing them, borrowing from the experimental design literature. The control is the baseline approach (e.g., single task training), while the treatment is the new approach or innovation (e.g., multitask or sequential training). Benefit is a quantitative measure of how good the outcome is, like accuracy, while cost measures what we pay to get it, such as dataset size or even dollars. Thus, cost equivalent curves can visualize how sequential training (the treatment) reduces data usage (the cost) compared to single task training (the control) when trying to achieve high accuracy (the benefit). Similarly, cost equivalent curves could visualize how Gaussian process optimization reduces hyper-parameter evaluations compared to random search when trying to achieve low perplexity on a language modeling task.
To construct cost equivalent curves, the main assumption is that the cost and benefit have a continuous, strictly monotonic (most often increasing) relationship. For machine learning, this assumption is satisfied empirically when using measures like expected cross-entropy against parameters, data, and compute (Kaplan et al. 2020). Since the cost and benefit share a monotonic relationship, we estimate the costbenefit trade-offs using isotonic regression (Barlow et al. 1972). Concretely, we test the control and the treatment at a bunch of different costs, and measure the benefit. Then, we fit a curve to the control's results,f c , and a curve to the treatment's results,f t . Since the cost equivalent curve,ĝ, maps the control costs to the treatment costs achieving the same benefit, we can estimate it as: That is, we compose the inverse cost-benefit curve for the treatment with the cost-benefit curve for the control. The inverse is guaranteed to exist because we assumed that the cost-benefit trade-offs are strictly monotonic.
Our implementation uses isotonic regression as implemented in scikit-learn (Pedregosa et al. 2011). To estimate the inverse curve, we switch the inputs and the outputs in the regression. The code may be found at https://github.com/ allenai/rainbow.

B Datasets
Our empirical study investigates transferring common sense from multisets (dataset collections) to various end tasks. Section 3 presented a new multiset, RAINBOW, for common sense transfer. In this appendix, Appendix B.1 describes each end task we evaluated, Appendix B.2 expands on RAINBOW and the other multisets we tried, and Appendix B.3 details the knowledge graphs we used.

B.1 Tasks
Six tasks, αNLI, COSMOSQA, HELLASWAG, PIQA, SO-CIALIQA, and WINOGRANDE, compose RAINBOW, as discussed in Section 3. Our experiments also use all six of these datasets as end tasks. In addition, we evaluated on COMMONSENSEQA, JOCI, and CYCIC. Each dataset is described below: αNLI (Bhagavatula et al. 2020) challenges models to infer the best explanation 8 connecting the beginning and ending of a story. Concretely, αNLI presents models with the first and last sentences of a three sentence story. The model must choose among two alternative middles based on which provides the most plausible explanation.
COSMOSQA (Huang et al. 2019) tests models' reading comprehension by asking them to read in-between the lines. Each example presents a short passage along with a question dealing with commonsense causes, effects, inferences and counterfactuals drawn from the passage. To solve the task, models must choose the best answer among four candidates.
HELLASWAG (Zellers et al. 2019) takes a context sentence and generates multiple completions using a language model. The machine generated endings often break commonsense world understanding, making it easy for humans to distinguish them from the original ending. In addition, HELLASWAG uses adversarial filtering (Zellers et al. 2018) to select the three distractor endings only from among those difficult for models to detect.
PIQA (Bisk et al. 2020) probes models' physical commonsense knowledge through goal-oriented question answering problems. The questions often explore object affordances, presenting a goal (e.g., "How do I find something lost on a carpet?") and then offering two solutions (such as "Put a solid seal on the end of your vacuum and turn it on" vs. "Put a hair net on the end of your vacuum and turn it on"). Models choose the best solution to solve the problem. COMMONSENSEQA (Talmor et al. 2019) offers general, challenging, common sense questions in a multiple-choice format. By construction, each question requires fine-grained world knowledge to distinguish between highly similar concepts. In particular, COMMONSENSEQA crowdsources questions by presenting annotators with three related concepts drawn from CONCEPTNET (Speer, Chin, and Havasi 2017). The annotators then create three questions, each picking out one of the concepts as the correct answer. To increase the dataset's difficulty, an additional distractor from CON-CEPTNET as well as one authored by a human were added to each question, for a total of five options.
CYCIC 9 offers five-way multiple-choice questions that touch on both common sense reasoning and knowledge over topics such as arithmetic, logic, time, and locations.

JOCI (Zhang et al. 2017) (JHU Ordinal Commonsense
Inference) generalizes natural language inference (NLI) to likely implications. Each problem presents a context followed by a hypothesis. In contrast to traditional NLI which explores hard, logical implications, JOCI instead explores likely inferences from the context. Thus, each example comes with an ordinal label of the likelihood: very likely, likely, plausible, technically possible, or impossible. In contrast to Zhang et al. (2017), we treat the task as five-way classification and evaluate it with accuracy in order to make it uniform with other end tasks we explore.

B.2 Multisets
In addition to RAINBOW, we use two other multisets for transfer. All three are described below.

B.3 Knowledge Graphs
In addition to multisets, we explored common sense transfer from the following knowledge graphs in Section 4.4: CONCEPTNET (Speer, Chin, and Havasi 2017) combines both expert curated and crowdsourced knowledge from various sources into a graph of concepts and relations. A concept is a short natural language word or phrase, such as "water". Connecting concepts, there's a commonly used set of canonical relations like ATLOCATION. For example, CONCEPT-NET contains the triple: "water" ATLOCATION "river". CONCEPTNET contains a significant amount of information beyond common sense; however, the common sense subset tends to focus on knowledge about objects and things.
ATOMIC (Sap et al. 2019a) offers a rich source of knowledge about the relationships between events and common sense inferences about them. ATOMIC connects events described in natural language using relations that express things like pre-conditions, post-conditions, and plausible inferences based on the event. For example, ATOMIC contains the triple: "PersonX makes PersonY's coffee" OREACT "PersonY will be grateful", where OREACT denotes the patient's (PersonY's) reaction.

C Training and Evaluation
This appendix describes the technical details of our training and evaluation setup, to help reproduce our experiments. It has an encoder-decoder structure and is pretrained using a combination of masked language modeling (Devlin et al. 2019) and multitask training on a large collection of NLP datasets. As a text-to-text model, T5 frames every NLP problem as mapping input text to output text. All structural information in the input is linearized into a sequence of text, similarly to Radford et al. (2018), and all output is generated as a string when making predictions. For training, T5 uses teacher forcing (Williams and Zipser 1989), i.e. maximum likelihood; for testing, T5 greedily decodes the generated text. Thus, for T5 to solve a task, one must first apply some straightforward preprocessing to frame it as text-to-text. Appendix C.2 describes the preprocessing we performed in more detail. Lastly, T5 is available in several model sizes: small (60M parameters), base (220M parameters), large (770M parameters), 3B (3B parameters), and 11B (11B parameters). For more information on T5 and its pretraining, see Raffel et al. (2019).

C.1 Model and Implementation
Our experiments use the original implementation, code, and weights for T5, which are publicly available at https://github.com/google-research/text-to-text-transfertransformer. Our code uses the original T5 implementation unmodified, only extending it with our own dataset preprocessing, reading, and task mixing. For deep learning operations, the implementation uses tensorflow (Abadi et al. 2016). Our code is available at https://github.com/allenai/ rainbow.

C.2 Preprocessing
To model tasks as text-to-text, we need to convert their inputs and outputs into strings. Our preprocessing first prepends a string to each example signifying its dataset, e.g.
[socialiqa]: for the SOCIALIQA task. Next, it wraps each feature in XML-like brackets with a unique tag identifying the feature, then joins them all together with newline characters. Figure 6 depicts an example from WINO-GRANDE. Preprocessing for other tasks is similar. Figure 6: An example of the dataset preprocessing applied to an instance from WINOGRANDE.

C.3 Training and Hyper-parameter Tuning
Following Raffel et al. (2019), we converted all tasks to textto-text and used teacher forcing (Williams and Zipser 1989) as the training objective, with greedy decoding for predictions. Our implementation reused the training and evaluation code from the original T5 paper. For leaderboard submissions and test set evaluations, we built UNICORN off of T5-11B. For all other experiments, we used T5-LARGE except when experiments specifically explore the impact of size, in which case the model size was explicitly indicated.
Hyper-parameters which were not set manually were tuned via grid search. In general, the fixed hyper-parameters and the grid used for search depended on the group of experiments, as outlined below. All hyper-parameters not mentioned were identical to those used in Raffel et al. (2019).
Leaderboard Submissions For leaderboard submissions and test set evaluations, T5-11B was initially multitasked on RAINBOW with an equal task mixing rate for 25,000 gradient updates using different hyper-parameter combinations to produce the UNICORNs. We then trained each on the end tasks separately for 25,000 gradient updates, saving a checkpoint every 2,500. The 10 most recent checkpoints were kept for early stopping, using dev set accuracy to choose the best checkpoint for evaluation. The grid search explored learning rates of 4e-3, 2e-3, 1e-3, and 5e-4 as well as batch sizes of 16 and 32.
Investigatory Experiments Experiments which were not evaluated on the test set or submitted to a leaderboard used the T5-LARGE model as a starting point, unless explicitly noted otherwise (e.g., in experiments exploring the impact of model size). Training was carried out for 50,000 gradient updates, saving a checkpoint every 5,000 and keeping the 10 most recent. The batch size was fixed to 16. Grid search explored learning rates of 4e-3, 1e-3, and 2.5e-4. Depending on the specific experiment, other hyper-parameters were explored as well. For models trained on full datasets (rather than learning curves), we explored equal and dataset size-weighted mixing rates when multitasking. In sequential training, this meant that these rates were tried during the initial multitask training before training on the end task alone. For transferring knowledge graphs, we also explored predicting the subject-relation-object tuples in forward, backward, and bidirectional configurations. When producing learning curves, i.e. training the model on subsets of the full data, we used the equal mixing rate for all mixtures and the forward direction for knowledge graph transfer. Given the extensiveness of these experiments, we chose not to evaluate these models on the test sets to avoid test leakage; thus, reported results for these experiments are always the best score on dev.
For transfer techniques requiring two stages of training (i.e. multitask fine-tune and sequential training), we reused the hyper-parameters from the first stage of training in the second stage. For all tasks, we used accuracy as the evaluation metric.
To facilitate reproducibility and future research, we release results for all of our experiments, including hyperparameter tuning. Download the results at https://github. com/allenai/rainbow. These tables contain all model evaluations and all hyper-parameter combinations tried in any given experiment.

C.4 Hardware, Software, and Compute
All experiments were run on Google Cloud using two Google Compute Engine virtual machine (VM) instances communicating with various TPUs. Experimental results were saved into Google Cloud Storage. Each VM had 20 vCPUs with 75GB of memory and ran Debian 9 (Stretch).
One VM used Intel Skylake vCPUs while the other used Intel Haswell. Specific versions of libraries and other dependencies used are available and tracked in the code repository.
For hardware acceleration, we ran all the experiments using v3-8 TPUs when building off of T5-LARGE or smaller. For T5-SMALL and T5-LARGE we used a model parallelism of 8, while for T5-BASE we used 4. The T5-11B models were trained using TPU v2-256 and v3-256s with a model parallelism of 16. Training times usually took several hours per run, so we ran many experiments in parallel on the VMs using GNU Parallel (Tange 2011).

D Leaderboards
As discussed in Section 5, UNICORN achieves state-of-theart performance across a number of popular commonsense benchmarks. This appendix collects those results along with the leaderboards' previous state-of-the-art and other useful baseline submissions for comparison.       Hyper-parameters For each of the submissions, we used the following hyper-parameters. αNLI used a learning rate of 5e-4 and a batch size of 16. COSMOSQA used a learning rate of 2e-3 and a batch size of 32. HELLASWAG used a learning rate of 2e-3 and a batch size of 32. PIQA used a learning rate of 2e-3 and a batch size of 32. SOCIALIQA used a learning rate of 5e-4 and a batch size of 32. WINO-GRANDE-xs used a learning rate of 2e-3 and a batch size of 16, WINOGRANDE-s used a learning rate of 2e-3 and a batch size of 16, WINOGRANDE-m used a learning rate of 5e-4 and a batch size of 32, WINOGRANDE-l used a learning rate of 1e-3 and a batch size of 16, and WINOGRANDE-xl used a learning rate of 2e-3 and a batch size of 16. CYCIC had a learning rate of 5e-4 and a batch size of 32, while COMMONSENSEQA had a learning rate of 1e-3 and a batch size of 32.

E Experiments
This appendix provides additional figures illustrating the findings as well as tables for all the experiments. In addition, the code used to run these experiments may be found at https://github.com/allenai/rainbow, and the models, experimental results (in CSV format) and even more figures, may be downloaded there as well.

E.1 Transferring to the RAINBOW Tasks
These figures and tables use RAINBOW for the end tasks: Figure 7 A comparison of different multisets using multitask training Figure 8 A comparison of different multisets using sequential training Figure 9 A comparison of different multisets using multitask fine-tune training Figure 10 A comparison of transfer methods on GLUE Figure 11 A comparison of transfer methods on SUPER-GLUE Figure 12 A comparison of transfer methods on RAINBOW Table 12 Single task baselines using the full training data Table 13 The performance using transfer and the full training data Table 14 Single task learning curves

E.2 Transferring to Other Tasks
These experiments target COMMONSENSEQA and JOCI. Figure 13 A comparison of different multisets using multitask training Table 21 Single task baselines using the full training data Table 22 The performance using transfer and the full training data Table 23 Single task learning curves Table 24 Learning curves using transfer

E.3 Effect of Size
These experiments explore the impact of model size on transfer using COMMONSENSEQA as the target dataset. Figure 14 A comparison of transfer methods across different model sizes Table 25 Full task performance for the initial multitask models used in sequential training and multitask fine-tune training experiments comparing model size

E.4 Transferring Knowledge Graphs
These experiments explore transferring knowledge graphs via multitask training, using RAINBOW for the end tasks. Figure 15 A comparison of transfer from different knowledge graphs Figure 16 A comparison of transfer from different knowledge graphs when also multitasking with RAINBOW Figure 17 A comparison of transfer from ATOMIC with and without multitasking RAINBOW Figure 18 A comparison of transfer from CONCEPTNET with and without multitasking RAINBOW Figure 19 A comparison of transfer from both ATOMIC and CONCEPTNET with and without multitasking RAIN-BOW Table 28 The performance using transfer and the full training data                   Figure 15: A comparison of transfer from different knowledge graphs to the RAINBOW tasks using multitask training. Performance is dev accuracy. Figure 16: A comparison of transfer from different knowledge graphs and RAINBOW to the RAINBOW tasks using multitask training. Performance is dev accuracy. Figure 17: A comparison of transfer from ATOMIC to the RAINBOW tasks via multitask training when also and not also multitasking with RAINBOW. Figure 18: A comparison of transfer from CONCEPTNET to the RAINBOW tasks via multitask training when also and not also multitasking with RAINBOW. Figure 19: A comparison of transfer from both ATOMIC and CONCEPTNET to the RAINBOW tasks via multitask training when also and not also multitasking with RAINBOW.