{'fraction_non_alphanumeric': 0.044613194114855245, 'fraction_numerical': 0.013278718969893317, 'mean_word_length': 4.156301537479385, 'pattern_counts': {'":': 8, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Ask Me Anything: A simple strategy for prompting language models

Large language models (LLMs) transfer well to new tasks out-of-the-box simply given a natural language prompt that demonstrates how to perform the task and no additional training. Prompting is a brittle process wherein small modifications to the prompt can cause large variations in the model predictions, and therefore significant effort is dedicated towards designing a painstakingly"perfect prompt"for a task. To mitigate the high degree of effort involved in prompt-design, we instead ask whether producing multiple effective, yet imperfect, prompts and aggregating them can lead to a high quality prompting strategy. Our observations motivate our proposed prompting method, ASK ME ANYTHING (AMA). We first develop an understanding of the effective prompt formats, finding that question-answering (QA) prompts, which encourage open-ended generation ("Who went to the park?") tend to outperform those that restrict the model outputs ("John went to the park. Output True or False."). Our approach recursively uses the LLM itself to transform task inputs to the effective QA format. We apply the collected prompts to obtain several noisy votes for the input's true label. We find that the prompts can have very different accuracies and complex dependencies and thus propose to use weak supervision, a procedure for combining the noisy predictions, to produce the final predictions for the inputs. We evaluate AMA across open-source model families (e.g., EleutherAI, BLOOM, OPT, and T0) and model sizes (125M-175B parameters), demonstrating an average performance lift of 10.2% over the few-shot baseline. This simple strategy enables the open-source GPT-J-6B model to match and exceed the performance of few-shot GPT3-175B on 15 of 20 popular benchmarks. Averaged across these tasks, the GPT-J-6B model outperforms few-shot GPT3-175B. We release our code here: https://github.com/HazyResearch/ama_prompting


Introduction
Large language models (LLMs) are bringing us closer to the goal of task-agnostic machine learning , Bommasani et al., 2021. Rather than training models for new tasks, LLMs are being applied to new tasks out-of-the box. In this paradigm, termed in-context learning, LLMs are instead controlled via natural language task specifications, or prompts. A prompt is defined by a template, which contains placeholders for the description and demonstrations of the inputs and outputs for the task. Figure 1: AMA first recursively uses the LLM to reformat tasks and prompts to effective formats, and second aggregates the predictions across prompts using weak-supervision. The reformatting is performed using prompt-chains, which consist of functional (fixed, reusable) prompts that operate over the varied task inputs. Here, given the input example, the prompt-chain includes a question()-prompt through which the LLM converts the input claim to a question, and an answer() prompt, through which the LLM answers the question it generated. Different prompt-chains (i.e., differing in the in-context question and answer demonstrations) lead to different predictions for the input's true label.
This work instead considers aggregating the predictions of multiple effective, yet imperfect, prompts to improve prompting performance over a broad set of models and tasks. Given a task input, each prompt produces a vote for the input's true label, and these votes are aggregated to produce a final prediction. In pursuit of high quality prompting via aggregation, we face the following challenges: 1. Effective prompts: High quality prompts are a precursor to improvements from aggregation. We take the original prompts which yield near-random performance in  for two SuperGLUE tasks (CB, RTE). Generating multiple prompts in the same format and taking majority vote prediction across prompts has a minor effect (+4% for CB) and can even hurt performance versus the average prompt performance (-2% for RTE). Many proposals for improved prompts focus on a single task type and evaluate on a single model-family and/or size [Wei et al., 2022a, Jung et al., 2022. We need a structure for prompting that works across tasks and models.
2. Scalable collection: After identifying effective prompt formats, we need to obtain multiple prompts in these formats -these prompts will be used to collect votes for an input's true label. The original format of a task varies widely and prior works manually rewrite input examples to new formats in a task-specific manner [Mishra et al., 2021, which is challenging to scale. We need a scalable strategy for reformatting task inputs.
3. Prompt aggregation: Using the prompts above (for CB and RTE), we see 9.5% average variation in accuracy and that the Jaccard index over errors is 69% higher than if prompt errors were i.i.d. Majority vote (MV) is the primary unsupervised aggregation strategy in prior prompting work [Jiang et al., 2020, Schick andSchütze, 2021], but it does not account for either property, making it unreliable. We need a strategy that accounts for the varying accuracies and dependencies.
In this work, we propose ASK ME ANYTHING PROMPTING (AMA), a simple approach that surprisingly enables open-source LLMs with 30x fewer parameters to exceed the few-shot performance of GPT3-175B. In AMA: 1. We identify properties of prompts that improve effectiveness across tasks, model types, and model sizes. We study standard prompt-formats categorized by prior work  and find prompts that encourage open-ended answers ("Where did John go?") to be more effective than prompts that restrict the model output to particular tokens (e.g. "John went to the park. Output True or False?"). For instance, converting three SuperGLUE tasks (CB, RTE, WSC) from the original restrictive formats in  to open-ended formats provides a 72% performance improvement (Section 3.2). Given a task input, we find that a simple structure of (1) forming questions based on the input and (2) prompting the LLM to answer the questions applies quite generally and improves performance across diverse benchmark tasks.
2. We propose a strategy for scalably reformatting task inputs to the effective formats found in (1). We propose to transform task inputs to the effective open-ended question-answering format by recursively using the LLM itself in a fixed two step pipeline. We first use question()-prompts, which contain task-agnostic examples of how to transform statements to various (e.g., yes-no, cloze) questions and second use answer()-prompts that demonstrate ways of answering questions (e.g., concise or lengthy answers). Applying prompt-chains-answer(question(x))-gives a final prediction for the input x. 2 Chains are (1) reused across inputs and (2) different pairs of functional prompts can be combined to create variety. We apply the varying functional prompt-chains to an input to collect multiple votes for the input's true label. 3. We propose the use of weak supervision (WS) to reliably aggregate predictions. We find that the errors produced by the predictions of different chains can be highly varying and correlated. While majority vote (MV) may do well on certain sets of prompts, it performs poorly in the above cases. AMA accounts for these cases by identifying dependencies among prompts and using WS, a procedure for modeling and combining noisy predictions without any labeled data [Ratner et al., 2017, Varma et al., 2019. We apply WS to prompting broadly for the first time in this work, showing it improves the reliability of prompting with off-the-shelf LLMs and no further training. We find that AMA can achieve up to 8.7 points of lift over MV and that on 9 tasks, it recovers dependencies among prompts to boost performance by up to 9.6 points.
We apply our proposed prompt-aggregation strategy, AMA, to 20 popular language benchmarks and 14 open-source LLMs from 4 model families (EleutherAI [Black et al., 2021, Wang andKomatsuzaki, 2021, EleutherAI], BLOOM blo [2022], OPT , and T0 ) spanning 3 orders-of-magnitude (125M-175B parameters). Our proof-of-concept provides an improvement over the few-shot (k = 3) baseline by an average of 10.2% ± 6.1% absolute (21.4% ± 11.2% relative) lift across models. We find the largest gains are on tasks where the knowledge required to complete the task is found in the provided context and comparatively less on closed-book tasks (e.g., factual recall). Most excitingly, ASK ME ANYTHING PROMPTING enables an open-source LLM, which is furthermore 30x parameters smaller, to match or exceed the challenging GPT3-175B few-shot baseline results in  on 15 of 20 benchmarks. We hope AMA and future work help address painpoints of using LLMs Ré, 2022, Narayan et al., 2022] by improving the ability to proceed with less-than-perfect prompts and enabling the use of small, private, and open-source LLMs.

ASK ME ANYTHING PROMPTING
We propose ASK ME ANYTHING PROMPTING (AMA), a prompting approach that uses multiple imperfect promptsrather than one painstakingly crafted perfect prompt-and reliably aggregates their outputs. We describe and motivate AMA's prompt format (Section 3.2), how AMA scalably produces collections of prompts (Section 3.3), and AMA's aggregation method (Section 3.4).

Preliminaries
We consider supervised tasks, (X , Y), where x ∈ X is the example and y ∈ Y is the output. We have an unlabeled dataset D = {x i } n i=1 for which we wish to predict each y i . We apply LLMs to this task by using a prompt-a natural language prefix that demonstrates how to complete a task. A prompt consists of a prompt template, with placeholders for (1) zero or more in-context task demonstrations and (2) for the inference example x as shown in Figure 3. Given a prompt p, we use p : X → Y to refer the output of the prompted LLM which produces a predictionŷ = p(x). Specifically, the LLM runs inference on p with x substituted for the placeholder in the template.
We denote a collection of m prompts as P = [p 1 , p 2 , ..., p m ]. Given input D, we (1) apply a collection P to each x ∈ D and (2) aggregate their predictions, denoted as P(x) = [p 1 (x), . . . , p m (x)], using an aggregator function φ : Y m → Y to produce outputsŷ on each x. We can thus express the procedure via two key components we aim to understand, the prompts P and aggregator φ.
Running examples For the motivating observations in the rest of this section, we use three SuperGLUE [Wang et al., 2019] tasks-CommitmentBank (CB), Recognizing Textual Entailement (RTE), and Winograd Schema Challenge (WSC)-and the DBPedia and AGNews classification tasks [Zhang et al., 2015]. We evaluate over the GPT-J-6B model Wang and Komatsuzaki [2021]. CB and RTE require determining the vailidity of a statement is given some context (as in Figure 1), WSC requires outputting the subject corresponding to a given pronoun, and DBPedia and AGNews contain 14 and 4 classes respectively. We use as a running example: determine if the statement "John went to the park" is valid, given the context "John invited Mark to watch Jurassic Park with his family at the theater".
Simple baseline To provide some intuition on the challenges around effectively designing the two levers, P and aggregator φ, we start with a naïve baseline with off-the-shelf prompts and the unsupervised majority vote prompt aggregation strategy used in prior work [Jiang et al., 2020, Schick andSchütze, 2021]. We take the prompts proposed in  for GPT-3 and produce P with five prompts for each task by using different sets of in-context examples. Comparing majority vote (MV), the unsupervised aggregation strategy used in prior work, to the average performance of the prompts, MV gives 39.3% (+2.2%) for CB and 54.5% (-2%) for RTE. The delta from aggregating is minor and in the worst case, harmful. Ideally, we would expect that aggregation should lead to improvement by reducing noise, but we find that performance here is only comparable to the single prompt baseline.

Effective Prompt Formats
First, we explore what makes an effective prompt format towards improving the quality of P(x).
Standard prompt formats We ground our analysis in three standard categories of prompts used in prior work including , Sanh et al. [2022, inter alia.]: (1) questions that restrict the model output particular tokens ("John invited Mark to come watch Jurassic Park. Output True or False?"); (2) cloze-questions which ask the model to fill in the remaining text ("John invited Mark to come watch Jurassic _" and using the LLM to fill-the-blank, "Park"); and (3) traditional (yes-no, Wh) free-form questions ("Where did John invite Mark?"). We compare these three prompting formats and make the following observations: 1. Open-ended prompts appear to outperform restrictive-prompts. We first group the results in Brown et al. [2020] based on the format used for the task, along the above categorizations (see Figure 2). When scaling from GPT3-6.7B to GPT3-175B, we find that the relative gain is far lower on open-ended (cloze and traditional QA) formats vs. restricted formats. Next, CB, RTE, and WSC are originally formatted with restrictive-prompts in ,    (Left). Ablating the promptstyle using the GPT-J-6B model. We include calibration results Zhao et al. [2021] and the "-" indicates the method cannot be applied to the task (Right).
is aligned with the next-token prediction language modeling objective. We observe that more precise questions give larger lifts. For WSC the restrictive prompt form is: "The pronoun 'his' refers to "Mark" in the context. True or False?", given the context "Mark went to the park with his dog.". Reformatting to "What does 'his' refer to?" and evaluating whether the answer is "Mark" provides 38% lift (69.2% accuracy). Yet, further extracting the portion of the context that mentions the pronoun ("his dog"), reformatting ("Whose dog?") and prompting with precise questions gives 49.4% lift (74.7%).

2.
The use of open-ended questions over restrictive-prompts can increase the difficulty of mapping open-ended answers to valid output classes. For tasks with output spaces that are likely observed during pretraining (yes-no questions, sentiment classification), we see that the LLM naturally generates validŷ ∈ Y. For tasks with specialized output classes (i.e. multi-class classification), we need to map the answer to the open-ended question (e.g., "What is the document about?") to a valid output class. For example, given 'Personality and Mental Health ... is a quarterly peer-reviewed academic journal published by ...", we observe that the LLM typically outputs semantically correct summaries of the document topic, e.g. "journal". We find that inserting a step for the LLM to map the open-ended output "journal" to a valid category via the prompt "A 'journal' maps to category: written work" enables a 33.3% and 11.1% lift over the few-shot baseline on DBPedia (14-way classification) and AGNews (4-way) respectively.
Why is the QA prompt format effective? We analyze the LM pretraining corpus to better understand why the proposed QA prompt template may be effective. The EleutherAI models are trained on The Pile corpus Black et al. [2021], Wang and Komatsuzaki [2021], Gao et al. [2021]. Over a 2% random sample of the ∼200B token Pile data, we find that open-ended QA structures (i.e., which ask the model "Is . . . ?", "Who . . . ?") appear on the order of 1000× more frequently than the restrictive-prompt structures (i.e., which instruct the model to output "True or False", "Yes or No"). The prompt structures and frequencies are in Table 8.
When applying the few-shot restrictive prompts, we observe large imbalances in the F1-scores for different classes (Table 10). Therefore, we next ask if answering the restrictive prompts is challenging due to biases acquired during pretraining. We find in Pile that there are large imbalances between the frequencies of "yes" vs. "no", and "True" vs. "False" for instance, which may instil the biases and contribute to the low quality from restrictive-prompts. Detailed results of the Pile analysis are in Appendix F.
AMA's prompt format Motivated by our observations about the effectiveness of QA prompt structures, we proceed in AMA with a two-step prompting pipeline: (1) generating questions based on the input and (2) prompting the LLM to answer the generated questions. These prompts are effective, and to further improve performance we next turn to generating and aggregating over multiple prompt-outputs for each input. For intuition, different questions (with our running example: "Who went to the park?", "Did John go the park?", "Where did John go?") emphasize different aspects of the input and can provide complementary information towards reasoning about the answer. Manually generating multiple prompts per input is challenging, and so we study how to do this at scale in the following section.

Creating Prompt Collections at Scale
Our goal is to produce a collection of prompts, P, that can be applied to tasks at scale. To produce prompts in the effective open-ended question-answering format, our insight is to recursively apply the LLM itself using a chain of functional prompts, referred to as a prompt()-chain. We describe these prompts as functional because they apply a task-agnostic operation to all inputs in the tasks, without any example-level customization. We describe the two functional prompts used in AMA below. We use Figure 1 as a running example to explain each type. (b) answer(): q → a applies the question generated by (a) to the context of x to produce intermediate answers a (such as "No" or "theater"). The answer() prompts contain demonstrations of how to answer a question (optionally) given some input context. To create P for aggregation, AMA constructs different prompt()-chains where each unique prompt()-chain is a different view of the task and can emphasize different aspects of x. Inspired by  and Liu et al. [2021], we also vary chains through two key levers-the in-context demonstrations and the style of prompt questions-as shown in Figure 3. To vary the style of open-ended prompt questions, we construct question() and answer() prompts that produce and answer either Yes/No, Wh, multiple-choice, or cloze-questions.

Prompt Aggregation
To aggregate the prompt predictions P(x) into outputsŷ reliably, we apply tools from weak supervision, a powerful approach for learning high-quality models from weaker sources of signal without labeled data [Ratner et al., 2017]. We first describe properties of P(x) that illustrate when the simple baseline of majority vote may perform poorly. We then describe our aggregator φ WS , which explicitly identifies and then accounts for these properties.
Baseline observations We understand how to aggregate P(x) by presenting a set of observations on CB, RTE, and WSC. For each, we compare two baselines for constructing P: (1) P T : varying the prompt template with no overlap in the in-context examples, and (2) P E : varying the in-context examples for a fixed prompt template, all with | P | = 5. We observe the following properties on P: 1. Varied overall accuracies: While prompts in P E may seem more similar than those in P T , the gap between the best and worst p i ∈ P is large in both cases -12.1% for P E and 9.6% for P T . 2. Varied class-conditional accuracies [Zhao et al., 2021]: Beyond overall prompt accuracy, the average variance of class-conditional prompt accuracies is 9.7% across the tasks and baselines. 3. Highly-correlated outputs: Prompt predictions have dependencies among each other. The Jaccard index over error sets averaged across tasks is 42.2 for P E and 39.9 for P T . For reference, two prompts that produce i.i.d. errors and have 60% accuracy each would have a score of 0.25.
The three observations present challenges in aggregating predictions via simple approaches like MV. MV tends to do better than using one prompt, but it weights all prompts equally and treats them independently. Such an aggregation method may be sufficient over certain collections of prompts but is not reliable across general P that may exhibit the three properties we have observed.
AMA Aggregation Given the varied accuracies and dependencies among prompt()-chains, in AMA we draw on recent work in weak supervision [Ratner et al., 2017], which is able to account for the accuracy and dependency properties without relying on labeled data. We learn a probabilistic graphical model on Pr G,θ (y, P(x)) and define the aggregator as φ WS (x) = arg max y∈Y Pr G,θ (y| P(x)). G = (V, E) is a dependency graph where V = {y, P(x)} and E is an edgeset where (p i (x), p j (x)) ∈ E iff p i (x) and p j (x) are conditionally independent given y. θ are the accuracy parameters for P(x). Since we lack labeled data y, we cannot estimate G or θ directly from D, so our procedure is as follows: The key insight is that the inverse covariance matrix on V , Σ −1 , is graph-structured, meaning that Σ −1 ij = 0 iff p i (x) and p j (x) are conditionally independent. This property yields systems of equations on V from which we can recover dependencies and accuracies, without any training. WS hence improves the reliability of aggregation.

Information Flow in AMA
Before evaluating end-to-end quality, we look at a simple information theoretic metric to understand the contributions of the individual components -P and φ -in the prompting procedure.
Information flow metric Specifically, we examine the conditional entropy, H(y|ŷ), which measures the amount of uncertainty remaining in the true label y given a predictionŷ. Intuitively, H(y|ŷ) will be low whenŷ encodes information relevant to y. In our setting,ŷ = φ(P(x)) is dependent on the two components of the prompting procedure, the prompts P and aggregator φ. The following simple decomposition of H(y|ŷ) enables studying the contribution of each component: Through the first term H(y| P(x)), H(y|ŷ) depends on the quality and quantity of the individual prompts in P(x) (since H(y| P(x)) ≤ H(y|p(x))). A set of prompts that contains relevant information for y contributes to a low H(y|ŷ). The second term H(y|ŷ) − H(y| P(x)) shows that H(y|ŷ) depends on how the aggregation step compresses the information in P(x) to predictŷ. An aggregator φ that more accurately matches the true Pr(y| P(x)) reduces the information loss in the compression step.
Evaluation We use (1) to evaluate our proposed solution AMA both empirically and theoretically. First considering H(y| P(x)), in Figure 4 (Left) we observe AMA outperforms k-shot baselines with expected scaling in terms of both individual prompt()-chain quality (as shown by AMA No Agg) and their quantity.
Next we consider the gap term H(y|ŷ) − H(y| P(x)). It enables us to understand why MV is insufficient: it compresses information from P(x) according to a specific construction of Pr(y, P(x)), for which p i (x) ⊥ p j (x)|y for all i, j ∈ [m], and Pr(p i (x) = c|y = c) for c ∈ Y is a single better-than-random constant across i and c. When the true distribution is vastly different-as is common-this misspecification results in a large gap between the optimal H(y| P(x)) and H(y|ŷ MV ) in Figure 4 (Right). Weak supervision can improve φ over the standard MV baseline to reduce the information loss H(y|ŷ AMA ) − H(y| P(x)).
In addition to empirical measurements, we can provide a theoretical characterization for the information flow. In Appendix D, we express H(y|ŷ AMA ) in terms of the individual prompt accuracies under the standard weak supervision model (i.e., Ising model on y and P(x) [Ratner et al., 2018]).
There has been recent interest in how LLMs improve primarily along the three axes of parameter scale, training data, and compute , Hoffmann et al., 2022, Wei et al., 2022c. In Figure 4, as we increase the number of prompts to be aggregated, the conditional entropy reduces. Prompt aggregation may be another useful axis for understanding LLM scaling performance.

Results
We evaluate ASK ME ANYTHING PROMPTING on 20 popular language benchmarks used in , . We report results across 14 unique LLMs including 4 model families (EleutherAI [Black et al., 2021, Wang andKomatsuzaki, 2021], OPT , BLOOM, and T0 ) spanning 3 orders-of-magnitude in size (125M-175B). We aim to validate whether AMA provides consistent lift across diverse tasks (Section 5.1), works across model families (Section 5.2), and reliably aggregates the predictions across prompts (Section 5.3).
For AMA we use 3 − 6 prompt()-chains to generate predictions per input. We model the correlations between prompt-predictions per task, without using any labeled training data, to obtain the final prediction per example via weak supervision (WS). We report both the average performance over the prompt()-chains (QA) and with AMA's WS aggregation (QA + WS). We report QA + WS across 5 random seeds for the model. Model details and prompt()chains are in the Appendix. 3

Main Results
We report benchmark results in Table 1 comparing the open-source GPT-J-6B and few-shot (k ∈ [32..70]) GPT3-175B. We find that the open-source 6B parameter model exceeds the average few-shot performance of the GPT3-175B model on 15 of 20 benchmarks. Over the 20 tasks, AMA gives an average improvement of 41% over the 6B parameter model's few-shot (k = 3) performance to achieve this.
We find that AMA provides the most lift on tasks where all requisite knowledge is included in the task input (e.g., reading comprehension) and that largely rely on model's natural language understanding (NLU) abilities. The lift is lower on tasks that rely on the LLMs memorized knowledge (e.g. commonsense, closed-book). AMA can help close the gap on knowledge-intensive tasks. The closed-book WebQ task includes simple questions, where the answers are likely seen during pretraining. We find that using an open-ended prompt that asks the LM to generate relevant context, and then prompting the model to answer the original question using the generated context is effective. However, there are limitations as seen on NQ.
We similarly see limitations when tasks cannot rely on the latent knowledge. We observe a small performance gap between model sizes on RealTimeQA, which includes questions that have temporally changing answers that are less likely to be memorized. Similarly, for tasks requiring domain knowledge, e.g. the "Amazon Instant Video" class in the Amazon task, all model-sizes achieve near-0 performance. In such cases, information retrieval may help close the gap. The flexible LLM interface permits asking and answering questions over diverse knowledge sources such as databases or a search engine . We provide an extended error analysis

Evaluation across Models
Benchmark results We evaluate the lift from AMA over out-of-the-box few-shot performance across different sizes of four open-source LMs (EleutherAI, OPT, BLOOM, and T0) across 7 tasks (4 NLU, 2 NLI, 1 classification). In this analysis, we want to understand the effectiveness of AMA's prompt()-chains reformattings across models and report the average prompt performance over the 3-6 prompt()-chains used per task. EleutherAI, OPT, and BLOOM are GPT models, while T0 is obtained by explicitly fine-tuning a T5 LM [Raffel et al., 2019] on prompt-input-output tuples.
Excitingly, the AMA prompt()-chains apply quite generally. We see a 10.2% ± 6.1% absolute (21.4% ± 11.2% relative) lift on average across models and tasks (see Figure 5a (a)). We observe the absolute lift increases with model size and levels out, however we note that there are few-models per size grouping. The average absolute (relative) lift by model family (across tasks and sizes) is 11.0% (24.4%) for EleutherAI, 11.0% (23.4%) for BLOOM, and 11.9% (22.7%) for OPT, and 2.9% (8.3%) for T0. We hypothesize the lower lift on T0 arises because the model was fine-tuned on zero-shot prompts, which may compromise its in-context learning abilities.
Diagnostics for understanding AMA lift To further understand why models see different degrees lift, we create a set of diagnostic tasks that correspond to the steps in prompt()-chains. The diagnostics measure four basic operations -question generation, answer generation, answer selection, and extraction. For each operation, we create 1-3 tasks with 50 manually-labeled samples per task. See Appendix E for task details.
We measure the average performance across each operation across different sizes of models in the four families (EleutherAI, OPT, BLOOM, and T0). We group models and sizes into four buckets of T0 (3B parameters) and GPT models (< 1B, 1B, and 6 − 7B parameters). Figure 5b shows results where the buckets are ordered by their average AMA lift across the 7 tasks from Section 5.2, meaning T0 (3B) sees the least lift while 6 − 7B GPT models realize the most lift. We find that overall, models with higher performance across the four operations see more lift with AMA. T0 performs poorly on the generative tasks, indicating the importance of text and question generation for AMA.

Evaluation against other aggregation methods
We compare our WS aggregation approach with the standard unsupervised approach, majority vote (MV), on prompt()-chains. We find that AMA can achieve up to 8.7 points of lift over MV, and does not do worse than MV on  16 out of 20 tasks. On the remaining 4 tasks, we perform worse than MV by at most 1.0 points. We also examine the effect of modeling dependencies in WS. We find that on 9 tasks, our approach recovers dependencies in the data (rather than assuming conditionally independent P(x)), which improves performance by up to 9.6 points and an average of 2.2 points. We provide more details and evaluation against labeled data baselines in    compared to majority vote (MV) and weak supervision (WS) over 10 different prompt formats in Prompt-Source. When using the Prompt-Source prompts, the average lift across tasks is 3.6 points for MV and 6.1 points for WS.
Next, we evaluate T0 on zero-shot prompts from the public PromptSource , which are better aligned with how this model has been trained. Specifically, we take 10 unique PromptSource prompts for CB, WIC, WSC and RTE respectively, and find that aggregating with MV yields an average lift of 3.7 accuracy points and aggregating with WS gives an average lift of 6.1 accuracy points (see Table 2).

Conclusion
In this work, we introduce ASK ME ANYTHING PROMPTING which (1) scalably obtains multiple prompts given a task input and (2) combines the intermediate answers to these prompts using weak supervision to give the final prediction. The steps in AMA stem from our observations on the effectiveness of open-ended questions over restrictive prompts, and the ability to model the varying accuracies and dependencies across a collection of prompts using weaksupervision. Overall, AMA provides lift across four language model families and across model sizes ranging from 125M-175B parameters. Most excitingly, we find that AMA enables a 30x smaller LM to exceed the average performance of few-shot GPT3-175B averaged across 20 popular language benchmarks. Several LM applications involve private data or require operating over large amounts of data -for these applications, using APIs to access closedsource models or hosting large models locally is challenging. We hope the strategies in AMA and subsequent work help enable such applications.

Reproducibility Statement
We release prompts and code for reproducing all benchmark results for few-shot and AMA prompting, and our diagnostic evaluation splits here: https://github.com/HazyResearch/ama_prompting.

Ethics Statement
We intend for AMA to aid practitioners in their exploration and use of LLMs-especially smaller, open-source LLMs. However, we recognize that AMA could be used to perform harmful or unethical tasks. AMA is a proof-of-concept; it has error-modes and we recognize the inherent risks to using LLMs. Detailed discussions of these risks are in Bommasani et al. [2021], Weidinger et al. [2021].

A Experiment Details
We use A100 NVidia GPUs to run all experiments.

A.2 Metrics
For RealTimeQA, the reported GPT-3 performance in Kasai et al. [2022] is reported over the text-davinci-002 API endpoint. Given that all our GPT-3 evaluations are over davinci, we re-evaluate the GPT-3 performance on RealTimeQA using the davinci endpoint and the few-shot prompt from RealTimeQA 4 .
We follow the metrics used in . All tasks are scored using matching accuracy except for DROP/Re-alTimeQA that use text f1, WebQ/NQ that use span overlap accuracy, and MultiRC that uses f1a accuracy.

A.3 Weak Supervision
For each task, we use an unlabeled dataset constructed from the test set as well as 1000 samples from the training set (ignoring the labels). We run the structure learning part of the weak supervision algorithm (forĜ) with the default parameters from Varma et al. [2019]. If the recovered sparse matrix has all entries greater than 1, we pass in an empty edgeset to the next step of learningθ (e.g., data is too noisy to learn structure from); otherwise, we pass in the edge with the highest value in the sparse matrix.

B.1 BLOOM Model Results
In Table 3, we provide results using the BLOOM-7.1B parameter model over all 20 benchmarks. We observe consistent lift over few-shot performance using AMA, though the performance remains below that of the comparably sized GPT-  Table 4: Results from applying prompt aggregation via Majority Vote and Weak Supervision to 3 random few-shot (k = 3) prompts.
Here we apply no prompt reformatting to the proposed AMA QA template.
J-6B parameter model reported in Table 5.1. We note that the few-shot results for BLOOM 7.1B are also often lower than the few-shot results for GPT-J-6B.

B.2 AMA Ablations
Here we extend the observations in Section 3 on additional tasks.
We study the degree to which both prompt re-formatting and aggregation are required to achieve high quality. Specifically, we produce 3 few-shot prompts (each with a different set of k = 3 in-context prompt examples), prompt using each, and aggregate the results using majority vote and weak supervision. We reiterate that the proposed AMA QA reformatting is not applied. We find that aggregation alone leaves large performance gaps. Aggregation alone is useful compared to the average performance, however aggregation and re-formatting are both critical and complementary in yielding an effective prompting solution.

B.3 Weak Supervision Ablations
Comparison to other aggregation baselines Table 5 compares AMA's aggregation method against several other baselines for aggregating prompt()-chains, including majority vote. We compare against weighted majority vote (WMV), where we use labeled data to weight according to each prompt's accuracy by constructing φ WMV (P(x)) = m i=1 exp(−ηε i )1 {p i (x) = y}. ε i is the error of prompt p i on a training set of 1000 examples, and η is a temperature hyperparameter, for which we perform a sweep over [0.25, 0.5, 1, 2, 4, 8, 16, 32] using a 20% validation split. We also compare against a simple strategy of using the prompt that performs the best on the labeled set of data (Pick Best). Finally, AMA (no deps) is our method when we pass in an empty edgeset to the algorithm in Ratner et al. [2018].
Varying amount of additional data We study the effect of varying the amount of additional unlabeled training data that is used in learning the probabilistic graphical model on y, P(x). On three tasks (RTE, WSC, and AGNews) averaged over 5 runs, we run AMA with 100%, 50%, 20%, 10%, and 0% of the additional dataset while still evaluating on the fixed test dataset. Figure 6 shows AMA's accuracy versus the amount of additional unlabeled data used. We find that even without any of the additional data, average accuracy does not decrease on WSC or AGNews, and only decreases by 0.4 points on RTE, still outperforming GPT3-175B few-shot. This suggests that the additional data is not necessary for AMA's performance.
Latency of Weak Supervsion Over RTE, WSC, and AGNews, we find that WS (both learning the graphical model and aggregating outputs) takes an average of 13.0 seconds when dependencies are not modeled. When dependencies are modeled in RTE (as dependencies are ignored in WSC and AGNews because they both exhibit dense recovered structured matrices), the algorithm takes an average of 84.3 seconds to run. As a point of comparison, we include Table 6 which shows the time in seconds for running inference with the GPT-J-6B model on the same tasks. The latency introduced by running weak supervision is comparatively low.  Table 5: AMA Aggregation method ablation for the GPT-J-6B parameter model, as well as the number of prompt()-chains used for each task. For ReCoRD, and QA tasks (DROP, WebQs, RealTimeQA, NQ), we use 3 prompts each and use majority vote as our aggregation strategy reported in the (QA + WS) columns of Table 1 and Table 3.  : Performance on RTE, WSC, and AGNews averaged over 5 runs when using varying amounts of additional unlabeled training data for estimating Pr(y, P(x)) in WS.

Task
Number of Examples Total Inference Cost (seconds) RTE 277 8310 WSC 104 3141 AGNews 7600 53200 Table 6: Total inference cost in applying the AMA prompt chains to achieve the results in Table 5.1, using the GPT-J-6B model.

B.4 Additional AMA Baselines
Here we compare AMA to Self-Consistency Wang et al. [2022b], which is particularly relevant in that it also aggregates over multiple prompt outputs without requiring any additional supervised training. Self-Consistency builds on Chain-of-Thought prompting Wei et al. [2022a], which proposes to guide the LM to generate reasoning paths in addition to the final prediction. We use the exact prompts and overlapping benchmark tasks provided in the Appendix of Wang et al. [2022b], using GPT-J-6B and report the results in Table 7. For Self-Consistency, we use temperature based sampling as discussed in Wang et al. [2022b], using temperatures ∈ {0.0, 0.3, 0.5, 0.6, 0.7}.
Overall, we observe AMA outperforms Self-Consistency at this model scale. This agrees with the results in Wang et al. [2022b] and Wei et al. [2022a], which report limited performance improvements for small LMs (<10B).  Table 7: Comparison between Self-Consistency Wang et al. [2022b] and AMA using GPT-J-6B and the same number of prompts.

C Weak Supervision Algorithm
We briefly explain the weak supervision algorithm used for constructing φ WS . Weak supervision models learn the latent variable graphical model on the distribution Pr(y, P(x)) using the dataset D, and aggregate votes using the learned distribution by setting φ(x) = arg max y Pr(y|P(x)). Our key insight in our aggregation approach is to parametrize Pr(y, P(x)) so that we can capture variations in accuracy as well as dependencies if they exist. The overall procedure of our aggregation is in Algorithm 1. Formally, we model Pr(y, P(x)) as a probabilistic graphical model are not conditionally independent given y and the other prompt()-chains, then The algorithm uses P(x) and D to first learn the dependency structureĜ among prompts using the approach from Varma et al. [2019]. The key insight from that work is that the inverse covariance matrix Σ −1 over y and P(x) is graph-structured, meaning that Σ −1 ij = 0 iff p i (x) and p j (x) are conditionally independent given y. The graph structure means that the inverse covariance over just P(x) decomposes into sparse and low-rank matrices, which can hence be estimated together using RobustPCA [Candès et al., 2011], and the sparse matrix can be used to recover the graph. Next, the algorithm uses the recoveredĜ along with P(x) and D to learn the accuracies of the prompts with the approach from Ratner et al. [2018]. The key insight from that work is to use the sparsity of Σ −1 to construct a system of equations set equal to 0 that recover the latent accuracy parameters. Once the parameters of the distribution are learned, we can compute PrĜ ,θ (y|P(x)) and aggregate our predictions.

D Information-Flow Theoretical Result
In equation 1, we decompose H(y|ŷ) into H(y|P(x)) and H(y|ŷ) − H(y|P(x)). For AMA, suppose that the weak supervision algorithm exactly recovers Pr(y, P(x)). That is,ŷ AMA is drawn from Pr(·|P(x)). Then, the second term H(y|ŷ) − H(y|P(x)) can be thought of as an irreducible error corresponding to how much information about y is lost in converting P(x) into an i.i.d. y randomly drawn from Pr(·|P(x)). Since y is more likely to change values when this distribution has high entropy, the second term is correlated with our first term H(y|P(x)), the amount of randomness in Pr(y|P(x)). We thus focus on obtaining an expression for H(y|P(x)) in terms of individual prompt accuracies.
We assume that Y = {−1, 1}. We model Pr(y, P(x)) as a probabilistic graphical model with dependency graph G = (V, E), where V = {y, P(x)}. The density of Pr(y, P(x)) follows the following Ising model commonly used in weak supervision [Ratner et al., 2017, Fu et al., 2020: where Z is the partition function for normalization and {θ y , Each θ i can be viewed as the strength of the correlation between y and p i (x), while each θ ij can be viewed as the strength of the dependence between p i (x) and p j (x). We assume that θ y = 0, which corresponds to Pr(y = 1) = 1 2 .
We present our expression for H(y|P(x)). Define Θ = [θ 1 , . . . , θ m ] to be the vector of canonical parameters corresponding to the strength of correlation between y and each p i (x). Define µ = E [p i (x)], which can be written as 2 Pr(p i (x) = y) − 1, a notion of accuracy scaled to [−1, 1].
Note that the above form of the distribution is in terms of canonical parameters θ. This distribution can also be parametrized in terms of the mean parameters corresponding to θ, which are Theorem 1. Assume Pr(y, P(x)) follows equation 2 above. Then, the conditional entropy H(y|P(x)) can be expressed as The quantity being subtracted from H(y) corresponds to the reduction in entropy of y given that we observe P(x). Within this expression, there are two terms. First, Θ µ is correlated with how much signal each p i (x) contains about y. Note that this quantity is symmetric-if p i (x) is negatively correlated with y, it still provides information since both θ i and E [p i (x)y] will be negative. The second term, E P(x) log cosh Θ P(x) , is for normalization (otherwise, the first term can grow arbitrarily large with Θ). Note that this quantity is independent of θ ij , the interactions between prompts.
We now write Pr(P(x)), Pr(P(x)|y = −1) and Pr(P(x)|y = 1) according to our Ising model in equation 2. Let Pr(P(x)) = Pr(P(x), y = −1) + Pr(P(x), y = 1) Therefore, we have that Plugging this back into equation 4, we have Substituting in our definitions of Θ and µ give us our desired expression for H(y|P(x)).

E AMA Diagnostics
We present a suite of 8 diagnostic tasks, which can be categorized into four task types: question generation, answer generation, answer selection and extraction. We provided details about the tasks and scoring below.
Question Generation: We measure the ability of the model to transform a statement to a question. We construct 3 question generation tasks which evaluate the models ability to transform a statement to a yes/no question (see Question Generation (Yes/No)), transform a statement to a whquestion (see Question Generation (wh-)) and finally, transform a statement about a placeholder entity to a question about the placeholder (see Question Generation (@placeholder)). All question generation tasks are scored using the ROUGE score [Lin, 2004].

Input
Rewrite the statement as a yes / no question .

Output
Where does most of the light come from ?
Answer Selection: We construct 2 answer selection tasks which measure the model's ability to generate an answer that is faithful to a set of provided answer choices. Concretely, we measure the models ability to select object categories from a fixed set of options specified in the context (see Answer Selection (category)). Further, we measure the model's ability to complete a sentence when provided with a context and set of sentence completion candidates (see Answer Selection (completion)). In both tasks, an answer is marked as correct if the generated response is one of the candidates provided in the context.

Answer Selection (category)
Input Select the correct category .
" Answer Generation: We construct 1 answer generation task which measures the model's ability to generate candidate sentence completions given a context and portion of a statement (see Answer Generation). Here, a generated answer is marked as correct if the model generates 2 candidate answers.

Answer Generation
Input Output a list of unique alternatives for each example .
Example : Barrack Obama believes the :

List alternatives : -best novel is Harry Potter
Output -worst book is Harry Potter -United States is great Extraction: We construct 2 extraction tasks which evaluate the ability of the model to extract spans from a given context. The first, and easier task, tests the model's ability to extract an attribute value from a wikibio (see Extraction (Span)). The second, more difficult task, tests the model's ability to extract the sentence from the context that mentions a specified entity (see Extraction (Sentence)). For both tasks, we use the Text-F1 score introduced in SQuAD [ Rajpurkar et al., 2018].

Input
Context : Caracas , Venezuela ( CNN ) --It ' s been more than 180 years since Venezuelans saw Simon Bolivar ' s face . But the revolutionary leader ' s thick sideburns , bushy eyebrows and steely gaze popped out from behind picture frames Tuesday in new 3 -D images unveiled by President Hugo Chavez . Researchers used several software programs to reconstruct the face of the man who liberated Bolivia , Colombia , Ecuador , Panama , Peru and Venezuela from the Spanish crown . Scans of Bolivar ' s skeletal remains , which investigators exhumed two years ago , factored into their calculations . So did historical paintings , photos of restored uniforms Bolivar wore and images of middle -aged Venezuelans , officials said .
Extract the sentence containing " Simon Bolivar ": Output Caracas , Venezuela ( CNN ) --It ' s been more than 180 years since Venezuelans saw Simon Bolivar ' s face .

F Understanding The Effectiveness of the Question-Answering Template
We analyze the LM pretraining corpus to better understand why the proposed QA prompt template may be effective. The EleutherAI models are trained on The Pile corpus Black et al. [2021], Wang and Komatsuzaki [2021], Gao et al. [2021].
Prompt patterns We compute the frequency of regular expression matches that correspond to the restrictive prompts (i.e., which instruct the model to output "True or False", "Yes or No") versus open-ended questions (i.e., which ask the model "Is . . . ?", "Who . . . ?") in a 2% random sample of the˜200B token Pile corpus. The restrictive prompt-patterns appear frequently in the original GPT-3 prompts . The frequencies are in Table 8.
We observe that question patterns appear more frequently than the restrictive prompts. Further, we find several instances of yes-no questions followed by "yes" or "no", which mimics the AMA format (Table 9). Overall, we find that QA structured text appears much more frequently in the pretraining corpus, which may help explain why the language models perform better on QA.
Word frequencies When applying the few-shot restrictive prompts, we observe large imbalances in the F1-scores for different classes (Table 10). Therefore, we next ask if answering the restrictive prompts is challenging due to biases acquired during pretraining. Over the same Pile sample as before, the mean word count is 25.3 ± 7309 occurrences. We compute the frequency of individual words in the "restrictive" and "open-ended question" patterns from Table 8. This leads to two hypotheses about why QA prompts perform well: 1. First we see that there are imbalances between the occurrence of "yes" vs. "no", and "true" vs. "neither" for instance. This may bias the model towards certain answer choices. Indeed Zhao et al. [2021] also hypothesizes, but does not provide any analysis over the pretraining corpus, that pretraining may instill particular biases in the model. 2. The frequency of the words in the "question words" categories is typically an order of magnitude larger than those in the "restrictive words" category. We hypothesize that the representations for the "question words" will be the most context-specific, which is useful for the prompting tasks we consider. Findings in ? support this hypothesis -? finds that frequently occurring words (e.g. stop-words) have the most context-specific representations. In other words, for the more frequently occurring stop-words the embedding produced by the transformer-based LM changes more significantly depending on the co-occurring words in the context.   Overall, designing prompting templates for an LM based on analysis of the LM pretraining corpus may be a promising path forward for future work.

G Error Analysis
We bucket the common error modes of AMA into three categories: knowledge, instruction-following, and longcontext.
Knowledge errors. We find that AMA yields the most gains when the knowledge required to complete the task is explicitly provided in the context (e.g., reading comprehension, extractive QA), which is in line with the trends in Figure  Figure 7: Relative performance gain observed when scaling from GPT3-6.7B to GPT3-175B. Results are directly from  and are categorized by type of knowledge required for the task. 7. We find that AMA provides comparatively less lift on tasks where the model needs to (1) recall encoded factual knowledge or (2) apply common-sense / real-world knowledge to a given context. We provide concrete examples from the Natural Questions dataset (see Knowledge (Factual) below) in which the GPT-J-6B model wrongly answers the question due to a lack of latent factual knowledge. We additionally provide case-examples from the BoolQ dataset where the model's limited real-world knowledge limits its ability to correctly answer the questions where the model's failure to recognize that food that is smoked is cooked, leads it to incorrectly answer the question (see Knowledge (Commonsense) below).

Input
Question : what ' s the dog ' s name on tom and jerry Answer :

Prediction
The dog ' s name is " Fido "

Input
Passage : A Philadelphia roll is a makizushi ( also classified as a kawarizushi ) type of sushi generally made with smoked salmon , cream cheese , and cucumber . It can also include other ingredients , such as other types of fish , avocado , scallions , and sesame seed . Question : is the salmon cooked in a philadelphia roll Answer : Prediction false Ground Truth true Instruction-following errors. We find that on tasks with more restrictive output spaces (e.g., multi-way classification tasks), a common failure mode is to generate an answer that is not in the desired output space of the AMA prompt, despite being explicitly prompted to do so. In Listing 3 and 4, we provide sample instances from the DBPedia classification task where GPT-J-6B does not correctly map a descriptive adjective (e.g., automobile or singer) to a valid class specified in the prompt.

Input
Pick one category for the following text . Long-context errors. We find that the AMA question() functions struggle to generate accurate statement-question transformations when the input is long or contains complex sentence structures (e.g. compound sentences). We provide sample instances from the SuperGLUE record task where GPT-J-6B fails to transform a sentence with a placeholder subject to a question about the placeholder subject (see Long-context (question()) below). Additionally, we find that the AMA answer() functions struggle to extract the correct span in long contexts (greater than 6 sentences). We show a sample instance from the DROP QA task where GPT-J-6B fails to extract the correct span from the long provided context (see Long-context (answer()) below).

Ground Truth
Who ' s wrath could be felt around noon on Wednesday ?

Input
Context : Looking to avoid back -to -back divisional losses , the Patriots traveled to Miami to face the 6 -4 Dolphins at Dolphin Stadium . After Carpenter ' s kickoff was returned from the 29 -yard line by Matthew Slater , the Patriots began their first possession at their own 40 -yard line . Cassel ' s first two passes were both completed for first downs , putting the Patriots in Dolphins territory and eventually their red zone . However , a holding penalty on Neal pushed the Patriots back 10 yards , forcing a 30 -yard Gostkowski field goal four plays later that gave the Patriots a 3 -0 lead . Following a Dolphins three -and -out , the Patriots ' second drive ended when a Cassel pass to Moss was bobbled by both Moss and cornerback Jason Allen to keep the ball in the air until Renaldo Hill intercepted it ; a 17 -yard return gave the Dolphins the ball at the Patriots ' 42 -yard line . On the next play , a 29 -yard David Martin reception moved the Dolphins into the Patriots ' red zone , where the Dolphins used their " Wildcat " formation on the next two plays ....

Question : Which team scored first ?
Prediction Patriots

H Datasets and Prompts
We evaluate over 20 datasets which fall into 4 categories: SuperGLUE (BoolQ [Clark et al., 2019]  SPACE . com -TORONTO , Canada --A second team of rocketeers competing for the #36;10 million Ansari X Prize , a contest for privately funded suborbital space flight , has officially announced the first launch date for its manned rocket . Category : SPACE . com -TORONTO , Canada --A second team of rocketeers competing for the #36;10 million Ansari X Prize , a contest for privately funded suborbital space flight , has officially announced the first launch date for its manned rocket . Summarize : the passage " Passage ":

Model Output
The passage is about a rocket .

answer()
Pick the correct category for the passage . SPACE . com -TORONTO , Canada --A second team of rocketeers competing for the #36;10 million Ansari X Prize , a contest for privately funded suborbital space flight , has officially announced the first launch date for its manned rocket . Summary : The passage is about a rocket . The summary " Summary " fits " Category ":

Is there information indicating whether Daniel Zolnikov is a good legislator ?
answer() Answer the question . If there is no evidence in the context , return " Unknown ".
Context : According to Biraben , the plague was present somewhere in Italy and affected 1 ,200 people . Question : Based on the context , Did the plague affect people in Europe ? Answer : yes , people in Italy , Europe Context : Policies aiming at controlling unemployment and in particular at reducing its inequality -associated effects support economic growth . Question : Based on the context , Is confidence a factor in increasing self -esteem ? Answer : unknown Context : The term " matter " is used throughout physics in a bewildering variety of contexts : for example , one refers to " condensed matter physics " , " elementary matter " , " partonic " matter , " dark " matter , " anti " -matter , " strange " matter , and " nuclear " matter . There is a little Shia community in El Salvador . There is an Islamic Library operated by the Shia community , named " Fatimah Az -Zahra ". They published the first Islamic magazine in Central America : " Revista Biblioteca Islamica ". Additionally , they are credited with providing the first and only Islamic library dedicated to spreading Islamic culture in the country .

Model Output
Is the community south of the United States ? answer() 33 Answer the question . If there is no evidence in the context , return " Unknown ".
Context : According to Biraben , the plague was present somewhere in Italy and affected 1 ,200 people . Question : Based on the context , Did the plague affect people in Europe ? Answer : yes , people in Italy , Europe Context : Policies aiming at controlling unemployment and in particular at reducing its inequality -associated effects support economic growth . Question : Based on the context , Is confidence a factor in increasing self -esteem ? Answer : unknown Context : The term " matter " is used throughout physics in a bewildering variety of contexts : for example , one refers to " condensed matter physics " , " elementary matter " , " partonic " matter , " dark " matter , " anti " -matter , " strange " matter , and " nuclear " matter . Question : Based on the context , Is anti -matter made of electrons ? Answer : Unknown Context : There is a little Shia community in El Salvador . There is an Islamic Library operated by the Shia community , named " Fatimah Az -Zahra ". They published the first Islamic magazine in Central America : " Revista Biblioteca Islamica ". Additionally , they are credited with providing the first and only Islamic library dedicated to spreading Islamic culture in the country . Question : Based on the context , Is the community south of the United States ? Answer :

Model Output
Does this headline lead to more information that is behind a paywall ?

answer()
Answer the question . If there is no evidence in the context , return " Unknown ".
Context : According to Biraben , the plague was present somewhere in Italy and affected 1 ,200 people . Question : Based on the context , Did the plague affect people in Europe ? Answer : yes , people in Italy , Europe Context : Policies aiming at controlling unemployment and in particular at reducing its inequality -associated effects support economic growth . Question : Based on the context , Is confidence a factor in increasing self -esteem ? Answer : unknown Context : The term " matter " is used throughout physics in a bewildering variety of contexts : for example , one refers to " condensed matter physics " , " elementary matter " , " partonic " matter , " dark " matter , " anti " -matter , " strange " matter , and " nuclear " matter . Question : Based on the context , Is anti -matter made of electrons Context : Drinking in public --Drinking in public is legal in England and Wales --one may carry a drink from a public house down the street ( though it is preferred that the user requests a plastic glass to avoid danger of breakage and because the taking of the glass could be considered an offence of Theft as only the drink has been purchased ) , and one may purchase alcohol at an off -licence and immediately begin drinking it outside . Separately , one may drink on aeroplanes and on most National Rail train services , either purchasing alcohol onboard or consuming one ' s own . Question : is it legal to drink in public in london Answer : Yes Context : Harry Potter and the Escape from Gringotts --Harry Potter and the Escape from Gringotts is an indoor steel roller coaster at Universal Studios Florida , a theme park located within the Universal Orlando Resort . Similar to dark rides , the roller coaster utilizes special effects in a controlled -lighting environment and also employs motion -based 3 -D projection of both animation and live -action sequences to enhance the experience . The ride , which is themed to the Gringotts Wizarding Bank , became the flagship attraction for the expanded Wizarding World of Harry Potter when it opened on July 8 , 2014. Question : is harry potter and the escape from gringotts a roller coaster ride ? True or False ? Answer :

Gold Output
True BoolQ AMA prompt()-chain Example answer() Answer the question using the context .
Context : Tonic water --Tonic water ( or Indian tonic water ) is a carbonated soft drink in which quinine is dissolved . Originally used as a prophylactic against malaria , tonic water usually now has a significantly lower quinine content and is consumed for its distinctive bitter flavor . It is often used in mixed drinks , particularly in gin and tonic . Question : does tonic water still have quinine in it ? Answer : yes Context : Northern bobwhite --The northern bobwhite , Virginia quail or ( in its home range ) bobwhite quail ( Colinus virginianus ) is a ground -dwelling bird native to the United States , Mexico , and the Caribbean . It is a member of the group of species known as New World quails ( Odontopho ridae ) . They were initially placed with the Old World quails in the pheasant family ( Phasianidae ) , but are not particularly closely related . The name '' bobwhite ' ' derives from its charac terist ic whistling call . Despite its secretive nature , the northern bobwhite is one of the most familiar quails in eastern North America because it is frequently the only quail in its range . Habitat degradation has likely contributed to the northern bobwhite population in eastern North America declining by roughly 85% from 1966 -2014. This population decline is apparently range -wide and continuing . Question : is a quail the same as a bobwhite ? Answer : yes Context : United States Department of Homeland Security --In fiscal year 2017 , it was allocated a net discretionary budget of $40 .6 billion . With more than 240 ,000 employees , DHS is the third largest Cabinet department , after the Departments of Defense and Veterans Affairs . Homeland security policy is coordinated at the White House by the Homeland Security Council . Other agencies with significant homeland security r e s p o n s i b i l i t i e s include the Departments of Health and Human Services , Justice , and Energy Question : is department of homeland security part of dod ? Answer : no

Context : Harry Potter and the Escape from Gringotts --Harry Potter and the Escape from Gringotts
is an indoor steel roller coaster at Universal Studios Florida , a theme park located within the Universal Orlando Resort . Similar to dark rides , the roller coaster utilizes special effects in a controlled -lighting environment and also employs motion -based 3 -D projection of both animation and live -action sequences to enhance the experience . The ride , which is themed to the Gringotts Wizarding Bank , became the flagship attraction for the expanded Wizarding World of Harry Potter when it opened on July 8 , 2014. Question : is harry potter and the escape from gringotts a roller coaster ride ? Answer : It is part of their religion , a religion I do not scoff at as it holds many elements which match our own even though it lacks the truth of ours . At one of their great festivals they have the ritual of driving out the devils from their bodies . First the drummers come on -I may say that no women are allowed to take part in this ritual and the ladies here will perhaps agree with me that they are fortunate in that omission . Question : no women are allowed to take part in this ritual True , False , or Neither ? True Modify the arachnids , said the researchers . Change their bodies and conditions , and you could get fibres like glass , still monofilament , but with logarithmic progressions of possibilities of strength and flexibility , and the ability to resonate light -particles or sound -waves undistorted , scarcely weakened over thousands of miles . Who said the arachnids had to be totally organic ? Question : arachnids had to be totally organic . True , False , or Neither ?

Model Output
Did arachnids have to be totally organic ?

answer()
Provide the answer to the question from the passage .
Passage : When Judy and Jack went to school , they got in trouble with their teacher for being late . I didn ' t think it was very fair . Question : Did she think it was fair ? Answer : No Passage : If inflation is occurring , leading to higher prices for basic necessities such as gas by 2 dollars . Do you think that inflation is good for society ? Question : Is inflation good for society ? Answer : Maybe Passage : Put yourself out there . The more time you spend dating and socializing , the more likely you will find a boyfriend you like . Question : Does socializing help you find a boyfriend ? Answer : Yes Passage : Modify the arachnids , said the researchers . Change their bodies and conditions , and you could get fibres like glass , still monofilament , but with logarithmic progressions of possibilities of strength and flexibility , and the ability to resonate light -particles or sound -waves undistorted , scarcely weakened over thousands of miles . Who said the arachnids had to be totally organic ? Question : Did arachnids have to be totally organic ? Answer : Gold Output false H.8 COPA Description: Casual reasoning dataset where task is to select the alternative that more plausibly has a causal relation with the premise. Wang et al. [2019] Train Size: 400, Test Size: 100

Input
Pick the more likely continuation to the following sentence .
Context : The truck crashed into the motorcycle on the bridge so The motorcyclist died .
Context : The customer came into the boutique because The window display caught her eye .
Context : The print on the brochure was tiny so The man put his glasses on .

Model Choices
-woman became famous so -photographers followed her Gold Output photographers followed her COPA AMA prompt()-chain Example answer() Pick the correct ending for the example .
Question : ( because ' she took medicine ' , because ' she got expelled ') My roommate was feeling better because ? Answer : ' she took medicine ' Question : ( because ' he does not practice ' , because ' he is fast ') Matt is not good at soccer because ? Answer : ' he does not practice ' Question : ( because ' she was smart ' , because ' she never did her homework ') The girl went to college and graduated with honors because ? Answer : ' she was smart ' Question : ( and so ' her family avoided her . ' , and so ' photographers followed her . ') The woman became famous and so ? Answer :

Model Choices
-woman became famous so -photographers followed her  Passage : TY KU -TY KU is an American alcoholic beverage company that specializes in sake and other spirits . The privately -held company was founded in 2004 and is headquartered in New York City New York . While based in New York TY KU ' s beverages are made in Japan through a joint venture with two sake breweries . Since 2011 TY KU ' s growth has extended its products into all 50 states . Summarize : the passage " Passage ":

Model Output
The passage is about a company .

answer()
Pick one category for the following text .

DROP Few Shot
Input Passage : As of the 2010 United States Census , there were 16 ,589 people , 6 ,548 households , and 4 ,643 families residing in the county . The population density was . There were 7 ,849 housing units at an average density of . The racial makeup of the county was 96.8% white , 0.7% black or African American , 0.6% American Indian , 0.2% Asian , 0.2% from other races , and 1.5% from two or more races . Those of Hispanic or Latino origin made up 0.6% of the population . In terms of ancestry , 23.4% were Germans , 22.3% were Americans , 13.6% were Irish people , and 11.0% were English people . Question : How many percent of people were not Asian ? Answer : unknown Passage : The health sector comprises 17 specialized hospitals and centers , 4 regional diagnostic and treatment centers , 9 district and 21 aimag general hospitals , 323 soum hospitals , 18 feldsher posts , 233 family group practices , 536 private hospitals , and 57 drug supply companies / pharmacies . In 2002 , the total number of health workers was 33 ,273 , of whom 6823 were doctors , 788 pharmacists , 7802 nurses , and 14 ,091 mid -level personnel . At present , there are 27.7 physicians and 75.7 hospital beds per 10 ,000 inhabitants . Question : What profession had more health workers , doctors or nurses ? Answer : nurses Passage : The exact number of peasant deaths is unknown , and even the course of events are not clear , because the government , to hide the size of the massacre , ordered the destruction of all documents relating to the uprising . Historian Markus Bauer mentions a greatly underesti mated official figure of 419 deaths , while an unofficial figure , circulated by the press and widely accepted , of about 10 ,000 peasants killed , has never been proven to be true . The same figure of 419 deaths was mentioned by Ion I . C . Bratianu in the Romanian Parliament . The data available to the Prime Minister Dimitrie Sturdza indicated 421 deaths between 28 March and 5 April 1907. Likewise , about 112 were injured and 1 ,751 detained . Newspapers patronized by Constantin Mille , Adevarul and Dimineata , gave a figure of 12 ,000 -13 ,000 victims . In a conversation with the British ambassador in Bucharest , King Carol I mentioned a figure of " several thousand ". According to figures given by Austrian diplomats , between 3 ,000 -5 ,000 peasants were killed , while the French Embassy mentioned a death toll ranging between 10 ,000 -20 ,000. Historians put the figures between 3 ,000 -18 ,000 , the most common being 11 ,000 victims . Question : Which organizations said the death toll to be beyond 10 ,000? Answer : Newspapers patronized by Constantin Mille Passage : Still searching for their first win , the Bengals flew to Texas Stadium for a Week 5 inter c on f er en c e duel with the Dallas Cowboys . In the first quarter , Cincinnati trailed early as Cowboys kicker Nick Folk got a 30 -yard field goal , along with RB Felix Jones getting a 33yard TD run . In the second quarter , Dallas increased its lead as QB Tony Romo completed a 4yard TD pass to TE Jason Witten . The Bengals would end the half with kicker Shayne Graham getting a 41 -yard and a 31 -yard field goal . In the third quarter , Cincinnati tried to rally as QB Carson Palmer completed an 18 -yard TD pass to WR T . J . Hou shmand zadeh . In the fourth quarter , the Bengals got closer as Graham got a 40 -yard field goal , yet the Cowboys answered with Romo completing a 57 -yard TD pass to WR Terrell Owens . Cincinnati tried to come back as Palmer completed a 10 -yard TD pass to H oushma ndzade h ( with a failed 2 -point conversion ) , but Dallas pulled away with Romo completing a 15 -yard TD pass to WR Patrick Crayton . Question : Which team scored the final TD of the game ? Answer : Answer the question . If there is no evidence in the context , return " Unknown ".

Gold Output
Context : According to Biraben , the plague was present somewhere in Europe in every year between 1346 and 1671 Question : Where was the plague present ? Answer : somewhere in Europe Context : Policies aiming at controlling unemployment and in particular at reducing its inequality -associated effects support economic growth . Question : What ' s one factor in increasing self -esteem ? Answer : Unknown Context : The term " matter " is used throughout physics in a bewildering variety of contexts : for example , one refers to " condensed matter physics " , " elementary matter " , " partonic " matter , " dark " matter , " anti " -matter , " strange " matter , and " nuclear " matter . Question : What is another name for anti -matter ? Answer : Unknown Context : Still searching for their first win , the Bengals flew to Texas Stadium for a Week 5 inter c on f er en c e duel with the Dallas Cowboys . In the first quarter , Cincinnati trailed early as Cowboys kicker Nick Folk got a 30 -yard field goal , along with RB Felix Jones getting a 33yard TD run . In the second quarter , Dallas increased its lead as QB Tony Romo completed a 4yard TD pass to TE Jason Witten . The Bengals would end the half with kicker Shayne Graham getting a 41 -yard and a 31 -yard field goal . In the third quarter , Cincinnati tried to rally as QB Carson Palmer completed an 18 -yard TD pass to WR T . J . Hou shmand zadeh . In the fourth quarter , the Bengals got closer as Graham got a 40 -yard field goal , yet the Cowboys answered with Romo completing a 57 -yard TD pass to WR Terrell Owens . Cincinnati tried to come back as Palmer completed a 10 -yard TD pass to H oushma ndzade h ( with a failed 2 -point conversion ) , but Dallas pulled away with Romo completing a 15 -yard TD pass to WR Patrick Crayton . Question : Which team scored the final TD of the game ? Answer : Model Output Passage : While this process moved along , diplomacy continued its rounds . Direct pressure on the Taliban had proved unsuccessful . As one NSC staff note put it , " Under the Taliban , Afghanistan is not so much a state sponsor of terrorism as it is a state sponsored by terrorists ." In early 2000 , the United States began a high -level effort to persuade Pakistan to use its influence over the Taliban . In January 2000 , Assistant Secretary of State Karl Inderfurth and the State Department ' s c o u n t e r t e r r o r i s m coordinator , Michael Sheehan , met with General Musharraf in Islamabad , dangling before him the possibility of a presidential visit in March as a reward for Pakistani cooperation . Such a visit was coveted by Musharraf , partly as a sign of his government ' s legitimacy . He told the two envoys that he would meet with Mullah Omar and press him on Bin Laden . They left , however , reporting to Washington that Pakistan was unlikely in fact to do anything ," given what it sees as the benefits of Taliban control of Afghanistan ." President Clinton was scheduled to travel to India . The State Department felt that he should not visit India without also visiting Pakistan .... Question : Based on the previous passage , What did President Clinton ' s visit with Pakistan include ? Is " Discussing Bin Laden " a correct answer ? Answer : Yes Passage : While this process moved along , diplomacy continued its rounds . Direct pressure on the Taliban had proved unsuccessful . As one NSC staff note put it , " Under the Taliban , Afghanistan is not so much a state sponsor of terrorism as it is a state sponsored by terrorists ." In early 2000 , the United States began a high -level effort to persuade Pakistan to use its influence over the Taliban  Passage : The Agencies Confer When they learned a second plane had struck the World Trade Center , nearly everyone in the White House told us , they immediately knew it was not an accident . The Secret Service initiated a number of security enhancements around the White House complex . The officials who issued these orders did not know that there were additional hijacked aircraft , or that one such aircraft was en route to Washington . These measures were precautionary steps taken because of the strikes in New York . The FAA and White House Teleco nf e re nc e s . The FAA , the White House , and the Defense Department each initiated a multiagency tele confer ence before 9:30. Because none of these teleconferences -at least before 10:00 -included ... Question : Based on the previous passage , To what did the CIA and FAA begin participating in at 9:40? Is " Coffee hour " a correct answer ? Answer : No Passage : What causes a change in motion ? The application of a force . Any time an object changes motion , a force has been applied . In what ways can this happen ? Force can cause an object at rest to start moving . Forces can cause objects to speed up or slow down . Forces can cause a moving object to stop . Forces can also cause a change in direction . In short , forces cause changes in motion . The moving object may change its speed , its direction , or both . We know that changes in motion require a force . We know that the size of the force determines the change in motion . How much an objects motion changes when a force is applied depends on two things . It depends on the strength of the force . It also depends on the objects mass . Think about some simple tasks you may regularly do . You may pick up a baseball . This requires only a very small force . Question : Based on the previous passage , Would the mass of a baseball affect how much force you have to use to pick it up ? Is " Yes " a correct answer ? Answer : Passage : Sara wanted to play on a baseball team . She had never tried to swing a bat and hit a baseball before . Her Dad gave her a bat and together they went to the park to practice . Sara wondered if she could hit a ball . She wasn ' t sure if she would be any good . She really wanted to play on a team and wear a real uniform . She couldn ' t wait to get to the park and test out her bat . When Sara and her Dad reached the park , Sara grabbed the bat and stood a few steps away from her Dad . Sara waited as her Dad pitched the ball to her . Her heart was beating fast . She missed the first few pitches . She felt like quitting but kept trying . Soon she was hitting the ball very far . She was very happy and she couldn ' t wait to sign up for a real team . Her Dad was very proud of her for not giving up . Question : Based on the previous passage , Who pitched the ball to Sara and where did it occur ? Is " Her dad did in the park " a correct answer ? Answer : yes Passage : The Vice President stated that he called the President to discuss the rules of engagement for the CAP . He recalled feeling that it did no good to establish the CAP unless the pilots had instructions on whether they were authorized to shoot if the plane would not divert . He said the President signed off on that concept . The President said he remembered such a conversation , and that it reminded him of when he had been an interceptor pilot . The President emphasized to us that he had authorized the shootdown of hijacked aircraft . The Vice President ' s military aide told us he believed the Vice President spoke to the President just after entering the conference room , but he did not hear what they said . Rice , who entered the room shortly after the Vice President and sat next to him , remembered hearing him inform the President , " Sir , the CAPs are up . Sir , they ' re going to want to know what to do ." Then she recalled hearing him say , " Yes sir ." She believed ... Question : Based on the previous passage , Why was the Secret Service ' s information about United 93 flawed ? Is " The Secret Service Didn ' t have access to FAA information " a correct answer ? Answer : no Passage : Patricia Cross and her boyfriend Larry Osborne , two students in a San Francisco school , become expelled for the publication of an off -campus underground paper . As a result , a philosophy professor , Dr . Jonathon Barnett , resigns his teaching position and decides to become an advocate for the c ounter cultur e youth movement and , specifically , the use of LSD . The hippies of the Haight -Ashbury district first see him as a hero and then as something even more . Dr . Barnett even makes an appearance on the Joe Pyne TV show to voice his support of the hippie community and the use of LSD . One scheming young man sees the opportunity to build Dr . Barnett as the head of a cult centered around the use of LSD . He hopes to earn profit from the users , Dr . Barnett ' s speeches known as '' happenings , '' and their lifestyles . At a massive LSD -fueled dance , Patricia begins to have a bad trip Which leads to an argument between her and Pat , ultimately splitting the couple up ... Question : Based on the previous passage , Why did Dr . Barnett resign from teaching ? Is " Patricia expulsion " a correct answer ? Answer : yes Passage : I wondered if that were my case --if I rode out for honour , and not for the pure pleasure of the riding . And I marvelled more to see the two of us , both lovers of one lady and eager rivals , burying for the nonce our feuds , and with the same hope serving the same cause . We slept the night at Aird ' s store , and early the next morning found Ringan . A new Ringan indeed , as unlike the buccaneer I knew as he was unlike the Quaker . He was now the gentleman of Breadalbane , dressed for the part with all the care of an exquisite . He rode a noble roan , in his Spanish ... Question : Based on the previous passage , Who is described as both buccaneer and cavalier ? Is " Quaker " a correct answer ? Answer : no Passage : What causes a change in motion ? The application of a force . Any time an object changes motion , a force has been applied . In what ways can this happen ? Force can cause an object at rest to start moving . Forces can cause objects to speed up or slow down . Forces can cause a moving object to stop . Forces can also cause a change in direction . In short , forces cause changes in motion . The moving object may change its speed , its direction , or both . We know that changes in motion require a force . We know that the size of the force determines the change in motion . How much an objects motion changes when a force is applied depends on two things . It depends on the strength of the force . It also depends on the objects mass . Think about some simple tasks you may regularly do . You may pick up a baseball . This requires only a very small force . Question : Based on the previous passage , Would the mass of a baseball affect how much force you have to use to pick it up ? Is " Yes " a correct answer ? Answer :

Model Output
Were security forces on high alert after a campaign marred by violence ? answer() Answer the question . If there is no evidence in the context , return " Unknown ".
Context : Jenna ' s 10 th birthday was yesterday evening and at least 10 of her friends attended the party . Question : Did 10 friends attend Jenna ' s party ? Answer : Unknown , at least 10 Context : The bullies attacked John when he was walking through the elementary school parking lot and then got sent to the teacher ' s office . Question : Did the bullies attack John in the teacher ' s office ? Answer : No , parking lot Context : WISS discovered a new monkey disease in a remote tribe in the Amazon rainforrest last week . It was highly contagious . Question : Did WISS discover a new disease ? Answer : Yes , new monkey disease Context : Security forces were on high alert after an election campaign in which more than 1 ,000 people , including seven election candidates , have been killed . Question : Were security forces on high alert after a campaign marred by violence ? Answer :

Input
Context : The University of Pennsylvania has been named America ' s top party school by Playboy in the first time the Ivy League institution has made the list . Believe it or not the magazine gave the top spot to the college by declaring that ' UPenn puts other Ivies to shame with its union of brains , brewskies and bros . ' In the magazine ' s ninth annual ranking of universities around the country , the University of Wisconsin -Madison scored the runner up slot . The University of Pennsylvania ( pictured ) has been named America ' s top party school by Playboy in the first time the Ivy League institution has made the list . The University of Wisconsin -Madison scored the runner up slot . Last year ' s winner West Virginia University slipped down to third place . It is the magazine ' s ninth annual ranking of universities around the country Answer : Playboy writes : ' Boasting a notorious underground frat scene that school officials have deemed a nuisance , these renegades pony up thousands of dollars ' worth of liquor for their parties -and competition among the houses means a balls -out war of debauchery .
Context : Garita Palmera , El Salvador ( CNN ) --She talks to the pictures as if they could make her voice travel thousands of miles and reach her son ' s ears . " Oh , my son ," Julia Alvarenga , 59 , says in a tender voice at her home in this coastal town . And then she says , "I ' m going to see him again ." The past two weeks have been an emotional roller coaster for the Salvadoran woman . First , she learned her son had been missing for 13 months . Then she was told he had turned up half a world away . And now she ' s getting news he might be back home soon .. It ' s been an emotional time for the parents of castaway Jose Salvador Alvarenga . His mother , Julia , said her son didn ' t keep up , and they didn ' t even know he was missing . " I would pray to God , and I won ' t lie to you , I was crying ," she says . For the excited residents of his town in El Salvador , Alvarenga is a hero Answer : Even though their son has yet to return home , he ' s already a celebrity in Garita Palmera and neighboring towns .
Context : ( CNN ) --Members of a well -known hacking group --according to a statement and Twitter messages --took credit Sunday for an online attack targeting San Francisco ' s embattled transit system . Anonymous --in a news release attributed to the group , and backed up by related Twitter pages --said it would take down the website of the Bay Area Rapid Transit System , known as BART , between noon and 6 p . m . PT Sunday . This is in response to the system ' s decision to cut off cellphone signals at " select " subway stations in response to a planned protest last week . " By ( cutting cell service ) , you have not only threatened your citizens ' safety , you have also performed an act of censorship ," a seemingly computer -generated voice --speaking over dramatic music and images --said in a video posted online Sunday afternoon .
" By doing this , you have angered Anonymous .". NEW : A video urges protesters Monday to wear red shirts and record the event . Statements attributed to Anonymous promised an online attack Sunday on BART .
MyBART . gov appears Sunday to have been hacked . The system said it was prepared for hacks , as well as a planned protest Monday Answer : " We ' re doing what we can to defend against any attack on the BART website ," the system said .. cable clutter and waste , European regulators say that Apple and other smartphone makers will be required to support a single common charging standard for all mobile devices as early as the fall of 2024. But Apple hates the idea ( shocker ) because that means about a billion devices will become obsolete . Article : 5 things to know for March 11: Ukraine , Pandemic , MLB , North ... If your day doesn ' t start until you ' re up to speed on the latest headlines , then let us introduce you to your new favorite morning fix . Sign up here for the '5 Things ' newsletter . ( CNN ) America , the " land of the free ," is getting quite costly . Prices for gas , food and housing --which are all necessary expenses --are spiking across the country . Gas prices have risen 38% over the past year , and rising prices in pandemic -related sectors , such as travel and dining , are also expected as the US recovers from the Omicron wave of Covid -19. Here ' s what you need to know to Get Up to Speed and On with Your Day . Article : Wi -Charge / consists of a transmitter and a receiver . Transmitter connects to a standard power outlet and converts electricity into infrared laser beam . Receivers use a miniature photo -voltaic cell to convert transmitted light into electrical power . Receivers can be embedded into a device or connected into an existing charging port . The transmitter automatically identifies chargeable receivers and start charging . Several devices can charge at the same time . According to Wi -Charge it can deliver several watts of power to a device at several meters away . The core technology is based on a " distributed laser resonator " which is formed by the re tr o re fl e ct or s within the Article : Mobile broadband / added in 2005. CDPD , CDMA2000 EV -DO , and MBWA are no longer being actively developed . In 2011 , 90% of the world ' s population lived in areas with 2 G coverage , while 45% lived in areas with 2 G and 3 G coverage , and 5% lived in areas with 4 G coverage . By 2017 more than 90% of the world ' s population is expected to have 2 G coverage , 85% is expected to have 3 G coverage , and 50% will have 4 G coverage . A barrier to mobile broadband use is the coverage provided by the mobile service networks . This may mean no mobile network or that service is limited to Article : Mobile edge computing / Combining elements of information technology and t el e c o m m u n i c a t i o n s networking , MEC also allows cellular operators to open their radio access network ( RAN ) to authorized third -parties , such as application developers and content providers . Technical standards for MEC are being developed by the European T e l e c o m m u n i c a t i o ns Standards Institute , which has produced a technical white paper about the concept . MEC provides a distributed computing environment for application and service hosting . It also has the ability to store and process content in close proximity to cellular subscribers , for faster response time . Applications can also be exposed to real -time radio access network ( RAN ) information . The key element is Question : To help reduce cable clutter and waste , which continent will soon require Apple and other smartphone makers to support a single common charging standard for all mobile devices ? Answer : ( CNN ) America , the " land of the free ," is getting quite costly . Prices for gas , food and housing --which are all necessary expenses --are spiking across the country . Gas prices have risen 38% over the past year , and rising prices in pandemic -related sectors , such as travel and dining , are also expected as Article 3: Wi -Charge / consists of a transmitter and a receiver . Transmitter connects to a standard power outlet and converts electricity into infrared laser beam . Receivers use a miniature photo -voltaic cell to convert transmitted light into electrical power . Receivers can be embedded into a device or connected into an existing charging port . The transmitter automatically identifies chargeable receivers and start charging . Several devices can charge at the same time . According to Wi -Charge it can deliver several watts of power to a device at several meters away . The core technology is based on a " distributed laser resonator " which is formed by the Article 4: Mobile broadband / added in 2005. CDPD , CDMA2000 EV -DO , and MBWA are no longer being actively developed . In 2011 , 90% of the world ' s population lived in areas with 2 G coverage , while 45% lived in areas with 2 G and 3 G coverage , and 5% lived in areas with 4 G coverage . By 2017 more than 90% of the world ' s population is expected to have 2 G coverage , 85% is expected to have 3 G coverage , and 50% will have 4 G coverage . A barrier to mobile broadband use is the coverage provided by the mobile service networks . This may mean no mobile network or that Text : in terms of execution this movie is careless and unfocused . Sentiment : negative Text : ... a pretentious and ultimately empty examination of a sick and evil woman . Sentiment : negative Text : the film 's plot may be shallow , but you ' ve never seen the deep like you see it in these harrowing surf shots . Sentiment : positive Text : a gob of drivel so sickly sweet , even the eager consumers of moore 's pasteurized ditties will retch it up like rancid creme brulee . Sentiment : Train Size: 1871, Test Size: 1871

Input
Given two possible next sentences A ) and B ) , choose the best next sentence to complete the story .
Answer with A or B .
Peter was excited to go to the Sanders rally in New Hampshire . As Peter entered the arena it was full of thousands of people . When Peter saw Bernie he cheered as loudly as possible . He felt thrilled to be there . A ) He couldn ' t wait to vote for him . B ) He was a staunch republican . Answer : He couldn ' t wait to vote for him .
My roommate was sick . She stayed home from work and school . She slept all day long . By the end of the day , she was feeling better . A ) She decided rest has helped . B ) She hoped she would soon be sick again . Answer : She decided rest has helped .
My aunt is a nurse . She often talks about long hours at work . Last week was especially bad . She didn ' t have a single day where she didn ' t work late . A ) It was easy work . B ) It was hard work . Answer : It was hardwork .
My friends all love to go to the club to dance . They think it ' s a lot of fun and always invite . I finally decided to tag along last Saturday . I danced terribly and broke a friend ' s toe . A ) My friends decided to keep inviting me out as I am so much fun . B ) The next weekend , I was asked to please stay home . Answer :

Model Choices
-My friends decided to keep inviting me out as I am so much fun . -The next weekend , I was asked to please stay home .

Gold Output
The next weekend , I was asked to please stay home . answer() Passage : My aunt is a nurse and she often talks about long hours at work . Last week was especially bad and she was constantly working many hours . Question : Was her work easy ? Answer : No , it was hard work .
Passage : My roommate was sick . She stayed home from work and school . She slept all day long and by the end of the day , she was feeling better . Question : Did the rest help her ? Answer : Yes , she slept and felt better .
Passage : Andy had always wanted a big kids bike . When he turned six Year ' s old he asked for a bike for his birthday . He did not know how to ride a bike . On Andy ' s birthday his mother gave him a bike . Question : Did he cry all night ? Answer : No , Andy was happy because he got a bike .
Passage : My friends all love to go to the club to dance . They think it ' s a lot of fun and always invite . I finally decided to tag along last Saturday . I danced terribly and broke a friend ' s toe . Question : Did I stay home the next weekend ?

Model Choices
-My friends decided to keep inviting me out as I am so much fun .
-The next weekend , I was asked to please stay home .

Gold Output
The next weekend , I was asked to please stay home .

H.18 WSC
Description: Task that requires readining a sentence with a pronoun and selecting the referent of that pronoun from a list of choices. Wang et al. [2019] Train Size: 554, Test Size: 104

Input
Passage : Mark was close to Mr . Singer 's heels . He heard him calling for the captain , promising him , in the jargon everyone talked that night , that not one thing should be damaged on the ship except only the ammunition , but the captain and all " his " crew had best stay in the cabin until the work was over . Question : In the passage above , does the pronoun " his " refer to Mark ? Answer : No Passage : Tom gave Ralph a lift to school so " he " wouldn ' t have to walk . Question : In the passage above , does the pronoun " he " refer to Ralph ? Answer : Yes Passage : This book introduced Shakespeare to Ovid ; it was a major influence on " his " writing . Question : In the passage above , does the pronoun " his " refer to Shakespeare ? Answer : Yes Passage : The large ball crashed right through the table because " it " was made of styrofoam . Question : In the passage above , does the pronoun " it " refer to the table ? Answer :

Gold Output
Yes WSC AMA prompt()-chain Example question() Extract the phrase containing the pronoun .
Passage : Jane ' s mom went to the shop to buy Jane a backpack for " her " first day of kindergarten . Extract : phrase containing " her ": " her " first day Passage : The musicians performed in the park and the crowd loved " them ". The crowd cheered for them . Extract : phrase containing " them ": crowd loved " them " Passage : Jeff gave his son some money because " he " wanted to buy lunch . Extract : phrase containing " he ": " he " wanted to buy Passage : The large ball crashed right through the table because " it " was made of styrofoam . Extract : phrase containing " it ":

Model Output
John Kasich is the current governor of Ohio .

answer()
Answer the question .
Context : The nearest airport to Palm Springs is Indio / Palm Springs ( PSP ) Airport which is 2.

Input
These aspects of civilization do not find expression or receive an interp retati on . Find the product of two numbers . Question : Is the word ' find ' used in the same sense in the two sentences above ? No Cut my hair . Cut the engine . Question : Is the word 'cut ' used in the same sense in the two sentences above ? No The pit floor showed where a ring of post holes had been . The floor of a cave served the refugees as a home . Question : Is the word ' floor ' used in the same sense in the two sentences above ? Yes An emerging professional class . Apologizing for losing your temper , even though you were badly provoked , showed real class . Question : Is the word ' class ' used in the same sense in the two sentences above ?
Gold Output no WiC AMA prompt()-chain Example answer() Give synonyms of the word in the sentence .
In " She heard the sound of voices in the hall ." , synonyms for the word " sound " are : -noise In " Enter the secret code ." , synonyms for the word " code " are : -password In " She acted in a play on Broadway " , synonyms for the word " play " are : -show In " An emerging professional class ." , synonyms for the word " ' class '" are : Model Output group