{'fraction_non_alphanumeric': 0.038608593788132695, 'fraction_numerical': 0.013162390381204953, 'mean_word_length': 4.759370314842578, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Forecast Aggregation via Peer Prediction

Crowdsourcing enables the solicitation of forecasts on a variety of prediction tasks from distributed groups of people. How to aggregate the solicited forecasts, which may vary in quality, into an accurate final prediction remains a challenging yet critical question. Studies have found that weighing expert forecasts more in aggregation can improve the accuracy of the aggregated prediction. However, this approach usually requires access to the historical performance data of the forecasters, which are often not available. In this paper, we study the problem of aggregating forecasts without having historical performance data. We propose using peer prediction methods, a family of mechanisms initially designed to truthfully elicit private information in the absence of ground truth verification, to assess the expertise of forecasters, and then using this assessment to improve forecast aggregation. We evaluate our peer-prediction-aided aggregators on a diverse collection of 14 human forecast datasets. Compared with a variety of existing aggregators, our aggregators achieve a significant and consistent improvement on aggregation accuracy measured by the Brier score and the log score. Our results reveal the effectiveness of identifying experts to improve aggregation even without historical data.


Introduction
Forecasting is one of the main areas where collective intelligence is frequently garnered. In crowd forecasting, a pool of human participants are invited to make forecasts on a set of prediction questions of interest and the solicited forecasts are then aggregated to obtain final predictions. Crowd forecasting has been widely applied in solving challenging forecasting tasks such as forecasting geopolitical events (Atanasov et al., 2016), predicting the replicability of social science studies (Liu et al., 2020a), diagnosing skin lesions  and labeling training sets for machine classifiers (Liu et al., 2012).
Aiming to more effectively leverage collective intelligence in forecasting, we focus on improving multitask forecast aggregation in this paper. We consider a minimal-information setting where each participant offers a single prediction to each forecasting question of a subset of total forecasting questions, and no other information such as participants' historical performance is available. By exploring only hidden information in participants' predictions over multiple questions, we develop a family of aggregation methods that robustly improves the accuracy of the final predictions across a variety of datasets.
The minimal-information setting requires the least effort to collect information and put almost no constraints on crowdsourcing workflow. Our methods can be used during the cold-start stage of long-term forecasting (Atanasov et al., 2016), where no event has been resolved yet to evaluate participants' performance. They can also serve as elegant benchmarks for developing more complex aggregators when additional information is available.
Our approach is to leverage peer forecasts to generate a proxy evaluation of each forecaster's performance that potentially positively correlates with her true performance. We call such proxy evaluations peer assessment scores (PAS). We then develop PAS-aided aggregators that build upon simple aggregators, such as mean. Our PAS-aided aggregators set larger weights in the simple aggregators on predictions from forecasters who obtain higher PAS.
The question then boils down to how to generate credible PAS evaluations. We are blessed by recent advances in the peer prediction literature. Peer prediction mechanisms are a family of reward mechanisms designed to use only peer reports on forecasting questions to motivate crowd forecasters to provide truthful or high-quality forecasts in the absence of the ground truth (Miller et al., 2005). While they are primarily developed for the purpose of forecast elicitation, Liu et al. (2020b) and Kong (2020) revealed theoretically that the rewards given by their mechanisms correlate positively with the prediction accuracy (defined using the ground truth) under certain conditions. Liu et al. (2020b) also showed empirical evidence of this correlation for several other peer prediction mechanisms.These mechanisms are potentially tools to use to construct the PAS-aided aggregators.
In this paper, we explore the use of five recently proposed peer prediction mechanisms (Radanovic et al., 2016;Shnayder et al., 2016;Witkowski et al., 2017;Liu et al., 2020b;Kong, 2020) as PAS. After showing their theoretical properties in recovering the forecasters' true performance, we thoroughly examine the empirical performance of PAS-aided aggregators built upon them. We employ 14 real-world human forecast datasets and two widely-adopted accuracy metrics, the Brier score and the log score. We compare the performance of these PAS-aided aggregators with four representative existing aggregators that neither require knowing the ground truth of resolved historical forecasting questions: the mean aggregator (Jose and Winkler, 2008;Mannes et al., 2012), the logit-mean aggregator, which is based on the idea of extremization of predictions (Allard et al., 2012;Satopää et al., 2014a;Baron et al., 2014), a statistical-inference-based aggregator (Liu et al., 2012), and the minimal pivoting aggregator, which is based on "surprising popularity." Palley and Soll, 2019) Our results reveal: 1) Though each of the above four existing aggregators has strong performance on specific datasets, none of them has consistent, robust performance across all datasets. 2) In contrast, our PAS-aided aggregators demonstrate a significant and consistent improvement in the aggregation accuracy compared to the four existing aggregators. 3) These PAS-aided aggregators adopt a very intuitive (explainable) and straightforward (generically applicable) strategy to incorporate PAS: select top forecasters according to their PAS and apply the mean or the logit-mean aggregator to the predictions of these selected forecasters. 4) Moreover, this improvement is observed when any one of the five peer prediction mechanisms is used as PAS, and there is no statistically significant difference found in the improvements when different PAS are used. 5) The above results demonstrate the possibility of discovering a smaller but smarter crowd in real-time forecast aggregation without accessing any ground truth outcomes.
We want to emphasize that aggregation without access to historical ground truth information is an incredibly challenging problem. One cannot expect that there is a universal aggregator that has the best performance on all datasets. There isn't. Instead, we hope to devise aggregators that perform well and robustly on different datasets. The significance of our work is three-fold. First, it provides a framework to select forecasts to achieve more robust and accurate aggregation.Second, our method can be used as a booster to aggregators in almost all multi-task forecast aggregation scenarios since it has minimal information requirements. Third, our work reveals a new and meaningful application of peer prediction methods -as scoring mechanisms to identify top experts and to improve forecast aggregation.

Related Work
Our work considers the multi-task forecast aggregation setting, where there is a set of (independent) judgement questions to forecast and each participant forecasts on multiple questions. A large part of the forecast aggregation literature considers the single-task setting, where all participants predict about a simple forecasting question. The methods and aggregators designed for the single-task setting are also often used in the multi-task forecast setting directly. Single-task aggregators include the mean, median, their trimmed variants (Galton, 1907;Clemen, 1989;Jose and Winkler, 2008;Mannes et al., 2012), the aggregators that extremize the mean predictions (Ranjan and Gneiting, 2010;Baron et al., 2014;Allard et al., 2012;Satopää et al., 2014a), and the "surprising-popularity"-based aggregators Palley and Soll, 2019;Palley and Satopää, 2020), which use the additionally collected participants' estimates about the other participants' forecasts to help aggregation. The aggregators proposed in our work also use single-task aggregators as building blocks. When there are multiple forecasting questions, the aggregation problem can also be viewed as learning a universal pattern between forecasters' predictions and the latent ground truth across forecasting questions. Therefore, statistical inference methods (Liu et al., 2012;Oravecz et al., 2014;Lee and Danileiko, 2014; are also customized and developed to aggregate forecasts in the multi-task setting. Our work includes both single-task aggregators and statistical-inference-based aggregators as benchmarks. We introduce more details about different aggregators in the benchmark selection part in Section 6.1. Our proposed aggregators use the heterogeneity of participants' expertise to improve aggregation accuracy. There is a large literature, including (Clemen and Winkler, 1986;Goldstein et al., 2014;Aspinall, 2010;Budescu and Chen, 2015;Satopää et al., 2014b), which explores this idea but in the case where forecasters' historical performance is available, or where the forecasting is conducted in a dynamic manner where forecasting questions are resolved sequentially, and the resolution can be used to aggregate unresolved questions. In contrast, we consider the scenario where no ground truth information is available, i.e., aggregated predictions are requested before any forecasting question is resolved. Wang et al. (2011) consider the same scenario. However, they assume that there exists a known logical dependence between the outcomes of different forecasting questions.
Our idea of peer assessment scores, which aims to measure a forecaster's prediction accuracy in the absence of ground truth information, is derived from multi-task peer prediction mechanisms (Prelec, 2004;Miller et al., 2005;Witkowski and Parkes, 2012;Radanovic et al., 2016;Kong et al., 2016;Agarwal et al., 2017;Witkowski et al., 2017;Goel and Faltings, 2019;Liu et al., 2020b), a family of mechanisms used to determine forecasters' rewards on multiple forecasting questions before any question resolves. For binaryvote judgement questions, Kurvers et al. (2019) proposed a measure of similarity of forecasters' votes, which is also empirically correlated with forecasters' true accuracy. In this work, we investigate the use of five representative peer prediction methods to generate PAS.

Setting
We consider the scenario with a set N of agents recruited to make forecasts on a set M of events (forecasting questions).
Events. We consider binary events (sometimes called tasks). 1 Each event i is represented by a random variable Y i ∈ {0, 1}, denoting the event outcome (ground truth). We assume that Y i is drawn from a Bernoulli distribution Bern(q i ) with an unknown q i ∈ [0, 1]. To illustrate, consider an event i as "Will Democrats win the 2024's election?" The outcome is either "Yes" (Y i = 1) or "No" (Y i = 0), and q i = 0.5 means that the outcome is random (at the time of forecasting) and the Democrats has 50% chance to win.
Agents. Each agent ( indexed by j) forecasts on a subset of events M j ⊆ M. M j could either be assigned by the principal or be constructed by agent j herself. We use N i ⊆ N to denote the subset of agents who forecast on event i. We use p i,j ∈ [0, 1] ∪ {∅} to denote the probabilistic prediction made by agent j on event i for Y i = 1, with p i,j = ∅ denoting agent j provides no forecast on event i. Meanwhile, we let p i = (p i,j ) j∈Ni and P = {p i,j } i∈M,j∈N .
The forecast aggregation problem.
The forecast aggregation problem is to design an aggregation function F : ([0, 1] ∪ {∅}) |M|×|N | → [0, 1] |M| , which maps the prediction profile P of all agents on all events to an aggregated prediction profile {q i } i∈M , whereq i ∈ [0, 1] is the aggregated prediction for event i. The design goal is to make the aggregated predictions as accurate as possible. The accuracy of predictions is evaluated against the corresponding ground truth of the forecasted events, which are expected to be revealed some time after the aggregation.
Our aggregators will use two popular existing single-task aggregators as building blocks: the mean (Mean) and the logit-mean (Logit) (Satopää et al., 2014a). Mean has empirically proved robustness (Jose and Winkler, 2008), while Logit extremizes the predictions of Mean and demonstrates significantly higher accuracy on some human forecast datasets (Satopää et al., 2014a). We introduce the weighted versions of the two aggregators that we will use as follows. For a single event i with a prediction profile p i and a weight vector (w j ) j∈Ni , we have • F Logit i (p i ) = sigmoid α |Ni| j∈Ni w j logit(p ij ) and α = 2 (Satopää et al., 2014a).
The Logit aggregator first maps probabilistic predictions into the log-odds space using the logit function, the inverse function of the sigmoid function. It then takes the weighted average and applies a scaling factor to  Prediction accuracy metrics The accuracy of forecasts is typically evaluated using the strictly proper scoring rules (SPSR) (Gneiting and Raftery, 2007). Two widely-adopted rules are the Brier score and the log score. We use them to evaluate our aggregators' performance in our experiments. For a predictionq i and ground truth Y i on an event i, we evaluate the two scores as follows: With above formulas, a lower scores refer to a higher accuracy. The Brier score ranges from 0 to 2. The log score ranges from 0.1 to 4.61. 3 An uninformative prediction of 0.5 receives a Brier score of 0.5 and a log score of 0.69 regardless of the event outcome.

Aggregation Using PAS
We now formalize the notion of peer assessment scores (PAS), and introduce our aggregation framework that uses PAS. We defer the introduction of concrete instantiations of PAS that lead to good aggregation performance into the next section. We list the abbreviations that we frequently use hereafter in Table 1. In short, and in different to the true accuracy that is evaluated against the ground truth, PAS assess a prediction against the other agents' predictions. Thus, unlike the true accuracy, PAS can be computed for all crowdsourcing forecasting scenarios, with no additional information (e.g., the ground truth) required. Formally, a peer assessment score on an event set M and an agent set N is a scoring function R : ([0, 1] ∪ {∅}) |M|×|N | → [0, 1] |N | that maps the prediction profile P of all agents on all events into a score s j for each agent j ∈ N . The score s j should reflect the average prediction accuracy of agent j.
Bearing this notion of PAS in mind, we introduce our aggregation framework. The intuition of our framework is straightforward: In aggregation, if we rely more on predictions from agents with higher accuracy indicated by PAS, we shall hopefully derive more accurate aggregated predictions. In general, we can incorporate PAS into an aggregation process via three steps: 1. Compute a PAS score s j for each agent j ∈ N .
2. Choose a weight scheme that weight agents' predictions based on the scores s j , j ∈ N .
3. Choose a base aggregator and apply the weight scheme to generate final predictions.
Each step features multiple design choices, which will influence the aggregation accuracy and can be customized case by case. In Step 1, there are multiple alternatives to compute PAS. Ideally, the computed PAS should reflect the true accuracy of agents. In Step 2, the weight scheme can be, for example, either ranking the agents by PAS and selecting a subset of top agents to aggregate (ranking & selection), or applying a softmax function to PAS to obtain weights.In Step 3, we can apply different base aggregators that can incorporate the weight scheme, such as weighted Mean or Logit.
Algorithm 1 PAS-aided aggregators 1: Compute PAS (using one of DMI, CA, PTS, SSR, PSR) based on all predictions. 2: Rank agents according to PAS. 3: For each event i, select the predictions from top max(10% · |N |, 10) agents who predict on that event, and run Mean or Logit aggregator on these predictions.
We call the aggregators following the above framework the PAS-aided aggregators. We present the detailed PAS-aided aggregators that we will test in this paper in Algorithm 1. In Step 1, we use five different peer prediction mechanisms (DMI, CA, PTS, SSR, and PSR) to compute PAS, which will be introduced in the next section. In Step 2, we choose the ranking & selection scheme rather than the softmax weight, as the former can be applied to any base aggregator and its hyper-parameter, the percent of top agents selected, has an straightforward physical interpretation. In our experiments, these two weight schemes show similar performance with best-tuned hyper-parameters. In Step 3, we use Mean and Logit as the base aggregator.

Peer Prediction Methods for PAS
Peer prediction mechanisms are a family of emerging reward mechanisms designed to incentivize crowd workers to truthfully report their private signals (e.g., probabilistic predictions or votes on the outcome) in the absence of ground truth information. These mechanisms can be expressed by a function R : ([0, 1] ∪ ∅) |M×N | → [0, 1] |N | that maps forecasters' prediction profile P to a reward R j for each forecaster j. The function R(·) is carefully designed so that an agent's expected reward according to her belief about others' reports (formed by her private signal) will be maximized when she reports truthfully.
While most peer prediction scores do not necessarily reflect prediction accuracy, we selectively review five peer prediction mechanisms in this section and provide theoretical support for using them as PAS -scores of these five mechanisms each correlate with accuracy of agents according to some metric. The core intuition of these peer prediction mechanisms to achieve truthful elicitation is to quantify and reward the correlations among participants' predictions that are associated with the ground truth of the forecasting questions, instead of rewarding the simple similarity between participants' predictions. As a result, forecasters with predictions containing more information about the ground truth tend to receive a better score in expectation. This property makes them ideal candidates to serve as PAS.
Two assumptions are often required for these mechanisms to work: A1. Events are independent and a priori similar, i.e., the joint distribution of agents' private signals and the ground truth is the same across events.
A2. For each event, agents' private signals are independent conditioned on the ground truth.
These two assumptions resemble the requirements for using statistical inference methods to infer the ground truth: there exists a consistent pattern between the ground truth and agents' predictions across tasks. The difference is that these two conditions do not restrict the pattern to follow some generative models specified by the inference methods. In the following paragraphs, we first introduce these five peer prediction mechanisms and then show why their rewards may correlate with agents' true prediction accuracy. We divide the five mechanisms into two categories.

Mechanisms recovering the strictly proper scoring rules (SPSR)
When SPSR are reoriented such that a higher score corresponds to higher accuracy, they can serve reward schemes to incentivize truthful reporting (Gneiting and Raftery, 2007). But they require the ground truth information to compute. Surrogate scoring rules (SSR) (Liu et al., 2020b) and proxy scoring rules (PSR) (Witkowski et al., 2017) are two peer prediction mechanisms that try to recover the SPSR from participants' reports, thus providing two methods to estimate the prediction accuracy of agents in the minimal information setting. Both mechanisms estimate a proxy of ground truth from participants' forecasts and assess their forecasts against this proxy. To introduce SSR and PSR, we use S(·) to denote an arbitrary SPSR.
Surrogate scoring rules (SSR) For a prediction p i,j from agent j, SSR randomly draws a binary signal Z from other agents' forecasts on the same task as the proxy to evaluate p i,j , with Z ∼ Bern The bias of Z to ground truth Y i can be represented by two error rates e 0 = P(Z = 1|Y i = 0) and e 1 = P(Z = 0|Y i = 1). Assumptions A1 and A2 guarantee that the error rates of Z for agent j are the same across different tasks. Based on this property, Liu et al. (2020b) provided an algorithm to accurately estimate e 0 and e 1 using participants' forecasts on multiple events. SSR then assess a prediction p i,j using a de-bias formula for S(·) to get an unbiased estimate for S(·) with Z. For prediction p i,j , we have . Proxy scoring rules (PSR) In constrast to SSR, PSR directly apply SPSR S(·) to an agent's forecast against a proxyŶ i of the ground truth to obtain the reward score, i.e., Witkowski et al. (2017) showed that as long as the proxyŶ i is unbiased to the ground truth, the proxy scoring rule gives an positive affine transformation of S(·), maintaining the incentive property. In practice, Witkowski et al. (2017) recommended using an extremized mean prediction as the proxy when there is no explicit unbiased proxy of ground truth available.

Mechanisms rewarding the correlation
Determinant mutual information mechanism (DMI) (Kong, 2020), correlated agreement (CA) (Shnayder et al., 2016), and peer truth serum (PTS) (Radanovic et al., 2016) are three mechanisms that reward agents by examining their forecasts' correlation to their peers'. Their core idea is to reward by a correlation metric that measures the agreement degree between agents' forecasts that are introduced through the ground truth, while excludes the agreement degree introduced by pure chance. In this way, an agent who independently manipulates her reports regardless the ground truth can only decrease her agreement with other agents. The computation of the expected reward under these three mechanisms for an agent j relies on the joint voting distribution between agent j and an uniformly randomly selected peer agent k. Given a prediction p i,j , agent j's vote on event i can be viewed as drawn from Bern(p i,j ). Thus, the joint voting probability of agent j voting u and agent k voting v for any u, v ∈ {0, 1} can be computed empirically aŝ where M j,k is the subset of forecasting questions answered by both agents. We useD j,k = dj,k u,v u,v∈{0,1} to denote the entire joint voting distribution of agent j and k. In the following paragraphs, we review how these three mechanisms reward agent j given the peer agent k.
Determinant mutual information mechanism (DMI) DMI measures the correlation using the determinant mutual information (Kong, 2020). Let M j,k , M j,k be two disjoint subsets of M j,k , and letD ,D be the joint voting distribution computed on these two subsets separately. DMI rewards agent j by an unbiased estimate to the squared determinant mutual information between agents j and k: where η is a normalization coefficient.
Correlated agreement (CA) CA rewards an agent j by whered j u = v∈{0,1}d j,k u,v is the marginal distribution of agent j reporting u estimated from the data. R CA j rewards the correlation by measuring the gap between the overall matching probability (represented byd j,k u,v ) and the matching probability caused by pure chance (represented byd j u ·d k v ).
Peer Truth Serum (PTS) PTS rewards agent j by the matching probability of her votes to the peer agent k's votes. PTS mitigates the effect of a match caused by pure chance via rewriting the matching probability under different vote realizations. Letp −j,u be the average marginal probability of voting u of all agents except j. PTS rewards agent j by

Peer prediction rewards and accuracy of agents
In this section, we formally show that the five peer prediction mechanisms reflect forecasters' true accuracy. First, SSR and PSR reflect the underlying accuracy of predictions due to the unbiasedness of their rewards w.r.t. the (affine transformation of) SPSR that they are built upon. As a direct corollary of their unbiasedness, we have the following.
1. Under Assumptions A1 and A2, SSR ranks the agents in the order of their mean SPSR that SSR is built upon asymptotically (|M|, |N | → ∞).
2. When there is an unbiased estimate of the ground truth and all agents are scored with the same unbiased estimate, PSR ranks the agents in the order of their mean SPSR that PSR is built upon asymptotically (|M| → ∞).
Second, the mechanisms, DMI, CA, PTS, reflect the accuracy of each agent because they essentially try to capture the informativeness of agents forecasts, i.e., the correlation between the agents' forecasts that is established through the ground truth instead of the pure chance. More specifically, we have the following proposition.
Proposition 2. Under Assumptions A1 and A2, and assuming agents report truthfully, the expected rewards of DMI, CA, PTS reflect a certain accuracy measure of agents. In particularly, 1. DMI ranks the agents in the order of their reports' squared determinant mutual information (Kong, 2020) w.r.t. the ground truth asymptotically (|M|, |N | → ∞).
2. CA ranks the agents in the order of their reports' determinant mutual information w.r.t. the ground truth asymptotically (|M|, |N | → ∞).
3. PTS ranks the agents in the inverse order of their signals' expected weighted 0-1 loss w.r.t. the ground truth outcome asymptotically (|M|, |N | → ∞), when the binary answer drawn from the mean prediction of all agents has a true positive rate and a true negative rate both above 0.5.
Item 1 in Proposition 2 follows straightforwardly from Theorem 6.4 in (Kong, 2020). We present the proofs for the items 2 and 3 in Appendix D. We note that mutual information does not directly imply accuracy in the binary case. For example, a random variable Y i = 1 − Y i contains all information w.r.t. the ground truth Y i . But Y i is clearly not an accurate prediction of ground truth Y i . However, when agents' forecasts p i,j are positively correlated to the ground truth Y i , i.e., agents' predictions are better than random guess, then the mutual information does rank forecasts in the correct order, i.e., ranking the perfect prediction (p i,j = Y i ) the highest and ranking random ones the lowest.

Empirical Studies
Our theoretical results suggest that the five peer prediction methods can effectively identify participants who predict more accurately than others under certain assumptions. In practice, however, it is often challenging or impossible to know to what extent these assumptions hold. Therefore, we conduct extensive experiments to study the performance of our PAS-aided aggregators. We use a diverse set of 14 real-world human forecast datasets and adopt two widely used accuracy metrics, the Brier score and the log score. We first introduce our experimental setup, then examine the effectiveness of PAS in selecting top performing forecasters, and finally present a comprehensive evaluation of our aggregators' performance. We first focus on binary events and then discussion our results on multi-outcome events in Section 6.4.    . These datasets vary in several dimensions, including dataset size, sparsity, topics, collecting environment, and participants' performance. Together they offer a rich environment for evaluating the performance of aggregators. The GJP and the HFC collected predictions about real-world issues involving geopolitics and economics via year-long online forecast contests. In these contests, forecasting questions were opened, closed, and resolved dynamically, and forecasters' accuracy can be evaluated using previously resolved questions and used to aggregate predictions of remaining open questions. In contrast, the MIT datasets are static prediction datasets, where participants predict on a set of questions all at once. The topics include the capital of states, the price interval of arts, and the diagnosis of skin lesions. The MIT datasets also contain additionally solicited predictions that participants made about other participants' predictions. This information enables one to apply the surprising-popularity-based aggregators.
Our paper focuses on the minimal-information aggregation setting. Therefore, we ignore the temporal information in the GJP and HFC datasets and only use each individual's final forecast on each forecasting question. 4 We also ignore the additional information solicited in MIT datasets when applying our aggregators, but use it for a surprising-popularity-based benchmark aggregator. We filter out participants with less than 15 predictions and questions with less than 10 answers from these datasets. This operation only removed a few forecasting questions in the HFC datasets with no sufficient predictions to make meaningful aggregation. We summarize the main statistics about the binary events of the 14 datasets after filtering in Table 2 and the multi-outcome events in Table 3. More details about datasets can be found in Appendix C.

Benchmark aggregators
In addition to the two base aggregators, Mean and Logit, which are widely-used in the minimal-information aggregation setting (Satopää et al., 2014a;Jose and Winkler, 2008), we also use two other types of aggregators as our benchmarks, the inference-based methods and the surprising-popularity-based methods.
• Inference-based methods contain a wide range of minimal-information multi-task aggregators. These methods establish parameterized models to characterize the latent features of forecasters such as their biases towards the ground truth probability and the variances in their beliefs. Then, they infer these parameters as well as the ground truth using the forecasts across all events. In this type of aggregators, we use the variational inference for crowdsourcing (VI) method as a benchmark. It is a go-to approach to aggregate predictions in the machine learning community. We use the estimate ground truth probabilities given by VI as its predictions. Details of VI are included in Appendix E. Other   sophisticated methods in this category include the cultural consensus model (Oravecz et al., 2014), the cognitive hierarchy model (Lee and Danileiko, 2014), and the multi-task statistical surprising popularity method (McCoy and Prelec, 2017) 5 . We will also compare to the performance these aggregators reported by McCoy and Prelec (2017) on the MIT datasets.
• Surprising-popularity-based methods are not minimal-information aggregators, but they represent a new trend of forecast aggregation Palley and Soll, 2019). They require forecasters to additionally predict other forecasters' predictions about the events of interest. Using this additional information, these methods can identify commonly shared information in participants' forecasts and avoid counting them multiple times in the aggregation. The typical aggregator in this category refers to the surprisingly-popular algorithm . We use a more recent variant, called the minimal pivot (MP) method, as our benchmark. It has a better performance in generating probabilistic predictions. It has a simple form: the aggregated prediction equals two times the mean of the participants' forecasts minus the mean of the participants' predictions about other participants' average prediction.
Median is another popular aggregator in the minimal information setting. In our test, its performance is always between the performance of Mean and Logit. Thus, we omit our results about median.

Implementation of PAS-aided aggregators
In our experiments, we evaluate 10 PAS-aided aggregators. Each PAS-aided aggregator uses one of the five peer prediction mechanisms (DMI, CA, PTS, SSR, PSR) to compute PAS and then incorporate the PAS into one of the two base aggregators (the Mean and Logit) using the rank&selection scheme. These PAS-aided aggregators have a single hyper-parameter-the number of top participants selected for each forecasting question. We set it to be the larger one of 10 and 10% percent of the total number of users. This hyper-parameter is shared among all PAS-aided aggregators on all datasets. Meanwhile, for SSR and PSR aggregators, we set the SPSR they are built upon as the metric SPSR. We use the output of the VI aggregator as the proxy used in PSR. 6 All these aggregators are described in Algorithm 1.

Smaller but smarter crowd
Before we dive into the comprehensive comparison between our PAS-aided aggregators and benchmarks, we first examine the effectiveness of PAS in identifying top forecasters and the influence of the number of top forecasters selected to the aggregation. Fig. 1 shows the average prediction accuracy of the top forecasters selected by the five PAS (DMI, CA, PTS, SSR, PSR) over the 14 datasets. For all five PAS, the average of the true mean Brier scores of the selected top forecasters steadily increases (from around 0.3 to around 0.45) when we gradually enlarge the selection range from top 5% to all forecasters. This result indicates that all five PAS scores effectively rank the forecasters in the order of their true performance. We also notice that at each level of top forecasters selected, the mean accuracy of top forecasters selected by different PAS is very similar. We further examine the overlap of these top forecasters. The result (Fig. 2) suggests that the sets of top forecasters selected by different PAS scores have considerable overlap, and among these overlapped forecasters, the portion of the actual top forecasters is also remarkable. For example, as shown in Fig. 2, around 50% of forecasters are common among the top 30% forecasters under different PAS scores, and in these common forecasters, 60% forecasters are the actual top 30% forecasters (because at the level of top 30%, 30% forecasters are shared by all 5 PAS together with the true Brier score). This result further confirms that the five PAS can identify true top performers and that they have similar abilities in doing so.
Next, we examine how the number of top forecasters selected by PAS influences the aggregation accuracy. Overall, we observe that the accuracy of the PAS-aided aggregators peaks at a certain top percent (usually at top 5% to top 20%) and outperforms the accuracy of the base aggregator that they are built upon. We illustrate this observation with dataset G2 in Fig. 3, which also shows the accuracy of a Brier-score-(BS)-aided aggregator. The performance of this BS-aided aggregator shows the "in hindsight" performance we could achieve if the peer assessment is as accurate as if we knew the ground truth. In this particular dataset, the PAS-aided aggregators perfectly recover this "in hindsight" performance of the BS-aided aggregator (Fig. 3).
Overall, these results confirm prior findings which show that there often exists a smaller but smarter crowd whose mean prediction outperforms that of the entire crowd (e.g. "superforecasters" (Mellers et al., 2015) and (Goldstein et al., 2014)). Our contribution is to demonstrate that we can identify this set of smarter forecasters using only their prediction information.

Forecast aggregation performance on binary events
In this section, we present our main experimental results-the aggregation performance of our 10 PAS-aided aggregators against the benchmark aggregators on binary events of the 14 datasets. Our extensive evaluation highlights the following findings: 1. The performance of the four benchmark aggregators varies significantly across datasets, confirming the difficulty of forecast aggregation in the minimal-information setting.
2. The PAS-aided aggregators not only have higher overall accuracy than the benchmarks but also perform more stably and robustly across datasets.
3. While the performance of the 10 PAS-aided aggregators is not statistically different, the Mean-based PAS-aided aggregators tend to have higher accuracy and lower variance than the Logit-based PAS-aided aggregators.
Our main results are shown in Table 4 and Table 5. Table 4 shows the accuracy of the 10 PAS-aided aggregators and the benchmark aggregators on each dataset under the Brier score. As can be seen, 9 out of 10 PAS-aided aggregators outperform the best of the benchmarks on at least 5 datasets, and the remaining one outperforms the best benchmark on 4 datasets. Furthermore, each of the 5 PAS-aided Mean aggregators outperforms the second-best benchmark on at least 12 out of 14 datasets. Moreover, no PASaided aggregator underperforms the worst benchmark on any dataset, with only one exception of the PSRaided Logit aggregator on dataset M1a. This is a significant improvement as we can see that though these benchmark aggregators are carefully designed for aggregating forecasts in the minimal information setting, none of them has stable performance across datasets. Table 5 provides the number of datasets on which one aggregator statistically outperforms the other for each pair of PAS-aided aggregators and benchmarks. Each of the 10 PAS-aided aggregators, especially the Mean-based PAS-aided aggregators, statistically outperforms each benchmark on at least 4 more datasets than it underperforms, with a maximum of 9 more datasets. Similar results are observed under the log scoring rule (Table 9, Appendix B and Table 5). Next, we give a more detailed review of the experimental results.
Performance of the benchmarks. The Logit aggregator performs better than the other benchmarks on the GJP and HFC datasets, but performs worse on the MIT datasets, while the Mean aggregator performs  in the other directions. This is likely because that the questions in MIT datasets are more challenging than those in the GJP and HFC datasets (e.g., see the correctness ratio of majority vote shown in Table 2), and the Logit aggregator, which extremizes the mean prediciton, further worsens the situation. VI predicts almost flawlessly on datasets M1b, M1c, but is outperformed by uninformative guess (predicting 0.5) on M2, M3, and M4a. This is likely because the accuracy of VI heavily depends on the extent to which the data follows the assumed generative model that VI uses to infer the ground truth. MP has a relatively stable performance on the MIT datasets, but on some of these datasets, it is outperformed by VI and Mean.
PAS-aided aggregators vs. Mean and Logit. As can be seen in Table 5, the PAS-aided aggregators outperform the Mean and the Logit aggregators with statistical significance on most datasets. Dataset H2 is the only exception where Mean and Logit are not outperformed by any PAS-aided aggregator under the Brier score. However, a closer look shows that the accuracy difference of these two aggregators in H2 is minimal (within 0.02). This advantage of the PAS-aided aggregators over the Mean and the Logit aggregators is because of the use of cross-task information when computing the PAS, i.e., the top forecasters are truly identified by these PAS using agents' forecasts on multiple tasks. These empirical results suggest that one can safely replace the Mean and Logit with the PAS-aided aggregators and expect an accuracy improvement in most cases (if a sufficient number 7 of predictions are collected from each forecaster to compute the PAS).

PAS-aided aggregators vs. VI and other inference-based methods
We notice that although VI ranks the worst in many datasets, the number of datasets on which VI statistically underperforms each PAS-aided aggregator is smaller than those numbers of the other benchmarks (Table 5). This is because VI tends to output extreme predictions (close to 0 or 1) and thus receives extreme accuracy scores (e.g., close to 0 or 2 under the Brier score), requiring more events to draw statistically significant conclusions. Also, as we have mentioned, the performance of VI varies significantly across different datasets (Table 4). If one is uncertain about whether the data follows the generative model assumed by VI, the PAS-aided aggregators (especially the SSR-/PSR-aided aggregators) are better choices. They perform much closer to VI than the other benchmark aggregators on datasets where VI makes almost perfect predictions (datasets M1b, M1c), and perform more stably on datasets where VI makes extremely wrong predictions (datasets M2, M3, M4a). (2017) reported the mean Brier score (with range [0,1]) of three other inference-based aggregators (the cultural consensus model, the cognitive hierarchy model and the multi-task statistical surprising popularity method) on MIT datasets (Table 10, Appendix B). Based on their reports, only the multi-task statistical surprising popularity method outperforms our PAS-aided aggregators on one more datasets than what VI does. However, this method requires forecasters to provide additional predictions beyond the predictions of the events of interest just as other surprising-popularity-based aggregators.

McCoy and Prelec
PAS-aided aggregators vs. MP MP generally performs better than other benchmarks on the 7 MIT 7 We will discuss this number in the next section.
Finally, we find no significant difference in the performance of PAS-aided aggregators that use different PAS. In particular, under the Brier score, no PAS-aided aggregator statistically outperforms another on more than three datasets if the same base aggregator is used. This is likely because different PAS have similar abilities in identifying the top forecasters as we have shown in Fig. 2.

Average performance across datasets
We present the mean and the standard deviation of the accuracy of our 10 PAS-aided aggregators and benchmarks over the 14 datasets in Fig. 4 (Concrete data can be found in Table 11, Appendix B). As can be seen, all PAS-aided aggregators have better mean accuracy under the Brier score than all benchmarks. In particular, the five Mean-based PAS-aided aggregators outperform all benchmarks with statistical significance  Table 6: The mean Brier score and the mean log score of different aggregators on multi-outcome events of 6 datasets. The best mean score among benchmarks on each dataset is marked by bold font. The mean scores of 10 PAS-aided aggregators that outperform the best of benchmarks on each dataset are highlighted in green; those outperforming the second best of benchmarks are highlighted in yellow; the worst mean scores over all aggregators on each dataset are highlighted in red.
(p<0.05) under both the Brier score and the log scoring rule. 8 Moreover, the five Mean-based aggregators also show much smaller variances than the Logit and VI aggregators under both accuracy metrics, suggesting that the Mean-based PAS-aided aggregators are more stable than these two benchmarks. Within PAS-aided aggregators, the Mean-based ones appear to be more accurate and stable than the Logit-based ones, while the differences are not statistically significant. We conjecture that as the PAS already select out the forecasters with more accurate predictions, the extremization provided by the Logit base aggregator no longer benefits for any accuracy improvement, but only increases the aggregation variance. These findings suggest that one can expect better accuracy and smaller performance variance when using PAS-aided aggregators instead of the benchmark aggregators. Moreover, the Mean-based PAS-aided aggregators, especially the Mean-based DMI-aided aggregator, are likely to produce the best aggregation outcomes. We also evaluated PAS-aided aggregators on smaller datasets that were sampled from the 14 original datasets. These datasets have 20 events and 30 or 50 participants. We observe similar improvements of the PAS-aided aggregators over the benchmarks. This result suggests that the PAS-aided aggregators may also mitigate the cold-start problem in long-term forecast aggregation settings, where only a small set of forecasts is available with no ground truth yet revealed. We present the details of this experiment in Appendix A.

Forecast aggregation performance on multi-outcome events
Our 10 PAS-aided aggregators can be extended to aggregate forecasts on multi-outcome events, because the 5 PAS scores and the two base aggregators can be extended to multi-outcome events (Satopää et al., 2014a;Radanovic et al., 2016;Shnayder et al., 2016;Witkowski et al., 2017;Liu et al., 2020b;Kong, 2020). However, the performance of these multi-outcome-event extensions may not be as good as their binary counterparts for two reasons. First, in the multi-outcome event settings, there are more latent variables to be estimated in the PAS scores, while the number of the samples (the multi-outcome events to forecast and the predictions collected) are usually smaller than those of binary events (Table 2 vs. Table 3). Second, the assumptions under which the PAS scores theoretically reflect the true accuracy of forecasters are more difficult to meet for multi-outcome events. Therefore, if we use these extended methods directly, the estimates of forecasters' performance may be noisy, leading to more noisy aggregated predictions.  The two-sided paired t-test for mean Brier score and mean log score of each pair of a PAS-aided aggregator and a benchmark on multi-outcome events of 6 datasets. The first integer in each cell represents the number of datasets where the PAS-aided aggregator achieves the significantly smaller mean score (with p-value<0.05), while the second integer in each cell indicates the number of datasets where the benchmark achieves the significantly smaller mean score.
A more practical alternative is to apply the PAS of forecasters estimated on binary events into the aggregation of multi-outcome events. In the GJP and HFC projects, agents face both binary events and multi-outcome events. Therefore, we can apply this approach on both GJP and HFC datasets. We present the statistics of multi-outcome forecasting questions in the GJP and HFC datasets in Table 3 and present the aggregation results and comparisons in Table 6 and 7. The results show a consistent and significant advantage of using the PAS-aided aggregators. The success in this approach also suggest that agents have consistent relative accuracy in making predictions on both binary events and multi-outcome events. In particular, on no dataset a benchmark outperforms a PAS-aided aggregation with statistical significance (the only exception is Logit v.s. CA-aided Mean on dataset H2).

Discussion and Future Directions
This paper demonstrates that the PAS-aided aggregators generally have higher aggregation accuracy across datasets than the four benchmark aggregators. Among the benchmarks, the Mean, Logit, and MP aggregators are single-task aggregators that generate the final prediction of an event using only the forecasts on that event. However, they were the top-performing aggregators in several real-world, multi-task forecasting competitions such as in the Good Judgement project (Jose and Winkler, 2008;Satopää et al., 2014a). The VI aggregator is a multi-task statistical-inference-based aggregator, which uses an inference method to infer the ground truth probability based on cross-task information. Our PAS-aided aggregators can also be viewed as a multi-task statistical-inference-based aggregator. The peer prediction methods used in the PAS-aided aggregators are inference-like methods that estimate forecasters' underlying expertise using all forecasts collected.
Using cross-task information in aggregation gives the PAS-aided aggregators advantages over the singletask benchmark aggregator. We can see that on datasets M1b and M1c, the three single-task benchmarks perform moderately well (with a mean Brier score around 0.3), while the other benchmark aggregator using cross-task information, the VI aggregator, has almost perfect predictions (with a mean Brier score close to 0). Our PAS-aided aggregators has similarly great performance on these two datasets as the VI aggregator. On the other hand, the PAS-aided aggregators appear to have more robust performance than the statisticalinference-based VI aggregator. For example, on datasets M2, M3, and M4a, where VI has much worse performance than random guesses, the PAS-aided aggregators still have moderate performance. Intuitively, statistical inference methods are sensitive to underlying properties of the data, i.e., the extent to which the assumed probabilistic model reflects the true pattern of the data. Unlike typical statistical-inferencebased aggregators, the PAS-aided aggregators do not directly infer the outcomes of the forecasting questions. Instead, they infer forecasters' expertise from cross-task predictions and then use the expertise information to adjust the base aggregator. This operation likely makes the PAS-aided aggregators more robust to the variation of the data.
Although the PAS-aided aggregators demonstrated significant accuracy improvement on datasets where individuals' overall performance is either good or poor and the number of forecasts collected per question is either high or low (GJP datasets and MIT datasets), we find their accuracy improvement is minimal on the HFC datasets, where the number of forecasts each forecaster made (< 40) is relatively small. This observation is consistent with the theoretical requirements for PAS scores to accurately estimate forecasters' true performance: Each forecaster has consistent accuracy across events, and each forecaster has made a sufficient number of predictions. Therefore, if an insufficient number of predictions has been made by each forecaster, the PAS scores may not reflect forecasters' factual accuracy well.
In addition, the five PAS scores that we tested in theory all rely on the assumption that the predictions of different forecasters are independent conditioned on the underlying event outcome to reflect the forecasters' true accuracy. Although the PAS-aided aggregators perform well on our 14 datasets, where the assumption is likely not hold strictly, one should still be careful about using the PAS-aided aggregators in scenarios where this assumption is saliently violated, for example, when forecasters are encouraged to discuss with each other before making predictions and when forecasters are machine predictors trained using similar data and methods.
In this paper, we take the first step to understand the possibility of using peer prediction methods to robustly improve the collective intelligence in prediction tasks. Our approach has the advantage of only requiring a minimal amount of information to be collected and placing almost no restriction on crowsourcing workflow. Thus, our methods have the potential of becoming a component of more interactive human-machine forecasting systems, where other techniques of boosting collective intelligence, such as teaming (Canonico et al., 2019), workflow design (Lin et al., 2012), promoting interactions (Bigham et al., 2015 and AI algorithms (Weld et al., 2015), are also present. From another perspective, the human-machine computation systems are now also developed for many complex tasks, such as image segmentation (Song et al., 2018) and article editing (Zhang et al., 2017). An important problem is that how we boost collective intelligence for solving these complex tasks. Our approach provides a way to potentially reduce this problem to how we can devise effective correlation metrics to capture the information quality of these responses. All above are interesting future research directions.     C More details about the datasets GJP datasets. GJP datasets (Ungar et al., 2012;Atanasov et al., 2016;GJP, 2016) contain four datasets about forecasts on geopolitical questions collected from 2011 to 2014. The dataset of each year differs in both the forecasting questions and the participant pools, and is denoted by G1 to G4 in our paper correspondingly. When collecting the forecasts, the participants were given different treatments: some were given probabilistic training, some were teamed up and allowed to discuss with each other before giving their own predictions, and some made predictions solely. Participants who demonstrated consistently high prediction accuracy across different forecasting questions in previous years were identified as "superforecasters" and were teamed up to participate in the forecast tournament in the following year (Mellers et al., 2015). The participants' prediction accuracy has also been shown to be influenced by different treatments (Atanasov et al., 2016).
HFC datasets. HFC datasets (IARPA, 2019) contain three datasets collected in 2018 with forecasting questions ranging from geopolitics to economics and environments. We use H1 to denote the dataset collected by the Hughes Research Laboratories (HRL), with participants recruited from Amazon Mechanical Turk (AMT) as H1. We use H2 to denote the dataset collected by IRAPA, with participants recruited from Amazon Mechanical Turk (AMT). Moreover, we use H3 to denote the dataset collected by IRAPA, with participants recruited via invitation and recommendation.

MIT datasets.
MIT datasets contain seven datasets (denoted as M1a, M1b, M1c, M2, M3, M4a, M4b ) collected for seven forecast behavior studies and for testing forecast aggregation methods. The forecasting questions range from the capital of states to the price interval of some artworks and some trivial knowledge. In the datasets, participants were asked to give binary (yes-or-no) answers to the forecasting questions instead of probabilistic predictions. Datasets M1c, M2, M3 also contain the confidence for the binary answers, which we directly interpret into probabilistic predictions of the favored binary answers when we aggregate the predictions. Moreover, all of the seven datasets contain participants' answers to an additional question for each forecasting question. This additional question asks the participants to estimate the percentage of other forecasters who choose the same binary answer as theirs. This information will be used by one of the benchmark aggregators we test. In particular, these seven datasets were collected to develop and evaluate information elicitation and aggregation methods on questions where the majority is likely to be wrong . Therefore, these datasets have a relatively low participants' performance.

D Missing Proofs
Proof. Proof of Theorem 2. The result about DMI is implied by Theorem 6.4 in (Kong, 2020). The result about CA can be proved in a similar way by observing that CA is asymptotically equivalent to determinant mutual information for binary events. For completeness, we present the proof for CA. We also present the proof for PTS below.
For CA: First, we introduce the determinant mutual information (Kong, 2020). Consider two discrete random variable X and W with the same support V. Let d(X, W ) = (d u,v ) u,v∈V be the joint distribution of X and W , where d u,v = Pr(X = u and W = v). Let d(X|W ) = (d u,v ) u,v∈V be the conditional probability matrix, where d u,v = Pr(X = u|W = v).
Definition 1. The determinant mutual information of two binary random variables X and W is | det(d(X, W ))|.
We denote the determinant mutual information of X, W as DM (X, W ) = | det(d(X, W ))|. We will involve the use of its two properties introduced below.
Proposition 3. Let X, X , W be three discrete random variables with the same support, and X is less informative than X w.r.t. W , i.e., X is independent of W conditioning X.
The information monotonicity is the key property for being a mutual information. Now, by Assumption A1 and the truthfulness assumption, we can consider the reported signal of agent j on a generic task as a binary random variable p j (p j ∈ {0, 1}). We denote the ground truth of the generic task as y and denote the joint distribution of agent j's reports and the ground truth as D j, * = (d j, * u,v ) u,v∈{0,1} , where d j, * u,v = Pr(p j = u and y = v). Similarly, let D j,k be the joint distribution of agent j's and agent k's reports. The empirical joint distributionD j,k is an unbiased and asymptotically consistent estimator of the true joint distribution D j,k . So asymptotically (|M | → ∞), we haveD j,k = D j,k . 10 Recall that CA compute the reward of agent j given a reference peer k as: where ∆ = (δ u,v ) u,v∈{0,1} , and δ u,v =d j,k u,v −d j u ·d k v . By trivial math, we have δ 0,0 = δ 1,1 = −δ 0,1 = −δ 1,0 and R DMI j = 2|δ 0,0 |. Further, asymptotically (|M| → ∞), |δ 0,0 | = d j,k 0,0 − d j 0 · d k 0 = | det(D j,k )| = DM (p j , p k ). Thus, we get that asymptotically (|M| → ∞), R DMI j = DM (p j , p k ) . Now, for another agent j = j, when k is also selected as her reference peer, we have asymptotically (|M | → ∞), ∝DM (p j , y) − DM (p j , y) This equation holds due to the relatively invariance of the determinant mutual information and the Assumption A2. This equation holds for any reference agent k = j, j . Thus, when agent j has a higher mutual information w.r.t. ground truth, she gets a higher reward than agent j' for any reference peer k = j, j . Asymptotically (|N | → ∞), with sufficient number of agents, the probability that agent j (j) are selected as agent j's (j 's) reference peer can be neglected. Therefore, the expected reward, with expectation taken over the reference peer selection, of CA rank the agents in the order of the determinant mutual information of agents' reports w.r.t. ground truth.
For PTS: By the counterpart argument in the proof for CA, under Assumption A1, we can treat the report p j as a random variable for a generic task with ground truth variable denoted as y. Letd u,v = j∈M d j, * u,v /|M| representing the joint distribution of a uniformly randomly picked report on a task w.r.t. the ground truth. Letd u =d u,0 +d u,1 , u ∈ {0, 1} be the marginal probability that an average agent reporting p j = 1. Further, let q v be the marginal distribution of y = v. We have q v = d j, * 0,v + d j, * 1,v , ∀v ∈ {0, 1} . Let E[R PTS j ] be the expected reward of agent j under PTS.

E Variational inference for crowdsourcing
Variational inference for crowdsourcing (VI), proposed in (Liu et al., 2012), is a computationally efficient inference method that builds a statistical model on agents' predictions over multiple questions to infer the ground truths of these questions. To make our paper self-contained, we present a sketch of VI, which mainly follows Section 3.2 of (Liu et al., 2012). VI consider the following statistical settings (assumptions): Agents provide binary predictions, i.e., p ij ∈ {0, 1} and have heterogeneous prediction abilities. Each agent j's prediction ability is characterized by a parameter c j , which is the correct probability of its predictions, i.e., c j = P(p ij = y i ), ∀i ∈ M j . Moreover, c j , ∀j are i.i.d. drawn from some beta distribution Beta(α, β) with an expectation no less than 0.5, i.e., E cj ∼Beta(α,β) ≥ 0.5, ∀j.
The goal of VI is to compute the marginal distribution of y i under the above statistical assumptions.
The marginal distribution is then used as the aggregated predictionq i for event i. Let δ ij = 1{p ij = y i }.

F Surrogate scoring rules
We illustrate how the error rates e 0 , e 1 of the noisy signal z for a particular agent j * on a particular task i * is estimated. We assume that the joint distribution of agents' reports and the ground truth on each event is the same. Let s i,j ∼ Bern(p i,j ) be a prediction signal for any agent j ∈ N on a task i ∈ M. z can be equivalently defined as a prediction signal uniformly randomly picked from set {s i,j } j =j * . At the same time, for each event i ∈ M, we uniformly randomly draw three prediction signals without replacement from {s i,j } j =j * and denote them by r i,1 , r i,2 , r i,3 . Given our assumption and with sufficiently number of agents, we know that the error rates of r i,1 , r i,2 , r i,3 w.r.t. the ground truth y i are the same to those of z for any i ∈ N . Let p 1 be the prior probability of y i = 1 for any i ∈ N . Therefore, we have the following three equations.
The left-hand-sides of above equations are the theoretical probabilities of a single, a double and a triple draws of z for an event i turning out to be all 1 given the error rates of z, respectively. The right-hand-sides are the observed real frequencies of a single, a double and a triple draws of predictions on an event turning out to be all 1, which can be computed using signal set {s i,j } i,j . These three equations hold exactly with infinite number of events and agents and hold approximately with finite number of events and agents. In non-trivial cases, there always exists a unique solution to these three equations while satisfies e 1 + e 0 < 1 and p 1 ∈ [0, 1]. The solution is given in Algorithm 2.
Algorithm 2 Estimation of error rates e 0 and e 1 for an agent j * on task i * Require: All predictions P Ensure: e 0 , e 1 1: Construct the prediction signal sets {s i,j } i,j =j * 2: Uniformly randomly select predictions without replacement z i , r i,1 , r i,2 , r i,3 from {s i,j } j =j * for all i ∈ N . 11 3: c 1 ← i∈N 1{ri,1=1} N ; c 2 ← i∈N 1{ri,1=ri,2=1} N ; c 3 ← i∈N 1{ri,1=ri,2=ri,3=1} For each event i, we can select the prediction from each agent j ∈ N i with probabilities proportional to 1 |M j | so as to achieve the uniformly randomly selection.