{'fraction_non_alphanumeric': 0.05026344717789318, 'fraction_numerical': 0.014252681854364884, 'mean_word_length': 4.1521335807050095, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Surrogate Scoring Rules and a Dominant Truth Serum

Strictly proper scoring rules (SPSR) are widely used when designing incentive mechanisms to elicit private information from strategic agents using realized ground truth signals, and they can help quantify the value of elicited information. In this paper, we extend such scoring rules to settings where a mechanism designer does not have access to ground truth. We consider two such settings: (i) a setting when the mechanism designer has access to a noisy proxy version of the ground truth, with known biases; and (ii) the standard peer prediction setting where agents' reports are the only source of information that the mechanism designer has. We introduce surrogate scoring rules (SSR) for the first setting, which use the noisy ground truth to evaluate quality of elicited information. We show that SSR preserves the strict properness of SPSR. Using SSR, we then develop a multi-task scoring mechanism -- called dominant truth serum (DTS) -- to achieve strict properness when the mechanism designer only has access to agents' reports. In comparison to standard equilibrium concepts in peer prediction, we show that DTS can achieve truthfulness in a multi-task dominant strategy. A salient feature of SSR and DTS is that they quantify the quality of information despite lack of ground truth, just as proper scoring rules do for the with verification setting. Our method is verified both theoretically and empirically using data collected from real human participants.


Introduction
Strictly proper scoring rules (SPSR) [1][2][3][4][5][6][7][8] have been developed to elicit private information (e.g. probability assessment of whether the S & P 500 index will go up next week) and evaluate the reported information for settings where the principal will have access to the ground truth (e.g. whether S & P 500 index actually went up) at some point. The score of an agent measures the quality of his prediction. Moreover, facing a strictly proper scoring rule, an agent strictly maximizes his expected score by truthfully revealing his prediction.
When it comes to information elicitation without verification (IEWV), settings where the principal does not have access to the ground truth and still wants to elicit private information (e.g. in peer grading, whether a student's homework solution is correct is unavailable as otherwise peer grading is not needed), existing mechanisms have achieved less in terms of both quantifying the quality of reported information and incentivizing truthful information revelation. An elegant family of mechanisms [9][10][11][12][13][14][15][16][17][18], collectively called peer prediction, has been developed for IEWV. Peer prediction leverages the correlation of agents' private signals and scores an agent's report based on how it compares with the reports from other agents. In other words, the peer prediction score of an agent measures the correlation of his report with those of others, but it doesn't reflect the report's true quality with respect to the ground truth. Furthermore, peer prediction incentivizes truthfully reporting at a Bayesian Nash Equilibrium (BNE). An agent reports truthfully

INTRODUCTION
when all other agents are truthful, but if some other agents are not truthful, then the agent can benefit from misreporting his information. 1 We focus on information elicitation without verification in this paper and ask: Can we design scoring mechanisms to quantify the quality of information as SPSR do and achieve truthful elicitation in dominant strategy for IEWV?
We provide a positive answer to this question for binary information elicitation. Two specific IEWV settings are considered. The first setting is when the principal has access to a random variable that is a noisy or proxy version of the ground truth, with known biases. The second setting is the standard peer prediction setting where agents' reports are the only source of information that the principal has.
For the first setting, we introduce surrogate scoring rules (SSR), which use the noisy ground truth to evaluate quality of elicited information, and show that SSR preserve the same information quantification and achieve truthful elicitation just as SPSR despite the lack of access to the ground truth. These surrogate scoring rules are inspired by the use of surrogate loss functions in machine learning [20][21][22][23][24] and they remove bias from the noisy random variable such that in expectation a report is as if evaluated against the ground truth.
Built upon SSR, in the second setting where the principal only has access to agents' reports, we develop a multi-agent, multi-task mechanism, the dominant truth serum (DTS), to again achieve information quantification and truthful elicitation in dominant strategy. The method relies on an estimation procedure to accurately estimate the average bias in the reports of other agents. With the estimation, a random peer agent's report serves as a noisy ground truth and SSR can then be applied to achieve information quantification as SPSR and truthful elicitation in dominant strategy. Finally, we evaluate DTS (i) on 7 human-generated judgement datasets and demonstrate that DTS scores (without access to the ground truth) rank reports similarly as the SPSR scores (with the ground truth), and (ii) on a large scale human prediction data collected from a forecast competition. Our results show that the DTS scores can approximate the true average Brier scores achieved by the participants well.
The rest of the paper is organized as follows. We will survey the most relevant results in the rest of this section. Section 2 lays out the preliminaries. Formulations of our elicitation problem is given in Section 3. We introduce surrogate scoring rules in Section 4. A dominant truth serum is proposed in Section 5 for eliciting information for the peer prediction setting. Our experimental results on a set of real human participant data are presented in Section 6. This paper is concluded with Section 7. We include the key and most proofs in the Appendix.

Related work
The most relevant literature to our paper is strictly proper scoring rules and peer prediction. Scoring rules were developed for eliciting truthful prediction (probability) [1,3,7,8,25,26]. For example, the pioneer works [1] proposed Brier scoring to verify the qualities of forecasts. A full characterization result is given for strictly proper scoring rules in [2,6,8]. The core idea of peer prediction is to score each agent based on another reference report elicited from the rest of agents, and to leverage on the stochastic correlation between different agents' information. This line of research started with the celebrated Bayesian Truth Serum work [9], where a surprisingly popular answer methodology is shown to be able to incentivize agents to truthfully report even when they believe they hold the minority answer (but more likely to be true in their own opinion). The seminal work [10] established that strictly proper scoring rule [8] can be adopted in the peer prediction setting for eliciting truthful reports (but the mechanism designer needs to know details of agents' model); a sequence of follow-up works have been done to relax the assumptions that have been imposed therein [12][13][14][15][16][17][18]. Our work complements both the strictly proper scoring rules literature by solving the case where the mechanism designer only has access to a noisy or proxy version of the ground truth or only unverified reports from agents, and the peer prediction literature by achieving truthful elicitation in dominant strategy.
As mentioned, our work borrows ideas from the machine learning literature on learning with noisy data [21,23,24,27,28]. From the high-level perspective, our goal in this paper aligns with the goal in learning from noisy labels -both aim to evaluate a prediction when the ground truth is missing, but instead a noisy signal of the ground truth is available. Our work addresses the additional challenge that the error rate of the noisy signal remains unknown a priori.

Preliminaries
Prior work on information elicitation typically considers one of two types of information: private signals and private beliefs (i.e. probabilistic predictions). In this paper, we develop mechanisms that apply to both cases.

Model of information
Suppose we are interested in eliciting information about a binary event y ∈ {0, 1} from a set of human agents, indexed by [N ] := {1, 2, ..., N }. Each of the N agents holds a noisy observation of y, denoting as s i . Agents' observations are conditionally independent: Pr[s i , s j |y] = Pr[s i |y] · Pr[s j |y]. We short-hand the following error rates: e 1,i := Pr[s i = 0|y = 1], e 0,i := Pr[s i = 1|y = 0], i.e., e 1,i , e 0,i are the error probabilities of agent i's observation for y. We do not assume homogeneous agents, that is we allow agents to have different e 1,i , e 0,i . The error rates can also model subjectivity in agents' private belief and observation. Based on signal s i , each agent can form a posterior belief about y, denoting as p i := Pr[y = 1|s i ]. When there are multiple tasks, we assume the realization of (across) the events has prior distribution P 0 := Pr[y = 0], P 1 = Pr[y = 1]. Suppose this prior distribution is non-trivial that 0 < P 0 , P 1 < 1. We further assume each agent's error rates, e 1,i and e 0,i , are homogeneous across tasks. 2 We consider eliciting either s i or p i from agents, 3 but we are not able to access the ground truth y to verify the reported information from agents. We call the elicitation of these two types of information as signal elicitation and prediction elicitation respectively. The literature on proper scoring rules has been primarily considering prediction elicitation, while the literature on peer prediction has focused on signal elicitation. To unify both types of information, we denote the information space as I, and each agent's information as I i . For signal elicitation, I i = s i and I = {0, 1}. For prediction elicitation, I i = p i and I = [0, 1]. Denote agent i's report to a mechanism as a i ∈ I. For signal elicitation and prediction elicitation, we have a i : s i → {0, 1} and a i : As a fact of the conditional independence among s i and above formulation, agents' reports satisfy conditional independence too: ∀i = j: P[a i , a j |y] = P[a i |y] · P[a j |y], ∀y. Note agents can simply choose to ignore their observations and "collude" by always reporting 0 or 1: P[a i = 1, a j = 1|y] = 1 and P[a i = 1|y] = 1, P[a j = 1|y] = 1 (always reporting 1) ⇒ P[a i , a j |y] = P[a i |y] · P[a j |y].

Strictly proper scoring rules
Many of our results build upon strictly proper scoring rules. We go through the basics of its setup. Strictly proper scoring rules were designed to elicit predictions, i.e. the p i 's. A scoring function S : where the expectation is taken with respect to the agent's belief Pr[y|s i ]. There is a rich family of strictly proper scoring rules, including Brier (S(p i , y) = 1 − (p i − y) 2 ), logarithmic and spherical scoring rules [8].
Though not enjoying much attention in the literature, the above idea of defining strictly proper scoring rules also applies to signal elicitation, that is to design a function S : For instance, with knowledge of the priors Pr[y = s], s ∈ {0, 1}, the following prior dependent output agreement scoring function is strictly proper: Lemma 1. 1/Prior scoring function is strictly proper.
We are not aware of other scoring functions for signal elicitation in the literature, except that we can adapt the strictly proper scoring rules aided peer prediction function to elicit signals, via replacing the reference signal with ground truth signal [10] -but this will require the mechanism designer to know the posterior distribution of agents' signals. For instance, to differentiate let us denote the scoring function for signal elicitation as S signal (), then for a strictly proper scoring function.
To unify prediction and signal elicitation, we will denote a strictly proper scoring rule for eliciting agent i's information I i as S(a i , y) : I × {0, 1} → R + .

Value of information with strictly proper scoring rules
To a certain degree, it's known that SRSR quantify the value/accuracy of reported predictions. We give a rigorous argument. First by representation theorem [2,6], we know that any strictly proper scoring function for prediction elicitation can be characterized using a corresponding convex function G which further gives us the following representation [8]: where e y is a vector with 1 for the component corresponding to outcome y and 0 elsewhere. D G is the Bregman divergence corresponding to function G: and G(p) is a subgradient of G. Taking expectation with respect to the true distribution of y we have which gives us the following proposition: For a strictly proper scoring rule S and its corresponding function G, a prediction p with smaller divergence "score" E y [D G (e y , p)] will have a higher score in S in true expectation.
Therefore E y [D G (e y , p)] captures the value of information for p under S. Intuitively speaking, the quantify characterizes how "far away" p is from the true distribution under divergence function D. When S is taken as the Brier scoring rule, the corresponding Bregman divergence is simply the quadratic function. Denoting the true distribution of y as p * , then E y [D G (e y , p)] = ||p * − p|| 2 , i.e., the prediction with a smaller l 2 norm (w.r.t. p * ) will enjoy a higher score in expectation.
For signal elicitation, there is very little characterization results. If we use strictly proper scoring rules aided peer prediction function [10], agent's scores are characterized again by D G evaluated at the posterior distributions of his reported signal. For 1/Prior scoring rule, we prove the following: Theorem 2. For 1/Prior scores, workers with smaller weighted sum-of-errors · e 1,i receive higher scores in expectation, i.e., e 1,i , e 0,i quantify the value of information for 1/Prior scores.
These quantifications can help the designer evaluate experts and rank them according to their performances.

Elicitation without verification
We formulate our elicitation problem in two settings, both without ground truth verification.

Model of elicitation with noisy ground truth
The first elicitation setting is when the mechanism designer has access to the realization of a binary random variable z ∈ {0, 1}, which is a noisy or proxy version of the ground truth with known bias. The bias of z is again captured by the error rates: e 1,z := Pr[z = 0|y = 1], e 0,z := Pr[z = 1|y = 0]. We cannot expect to do much if z is independent of y and hence assume that z and y are stochastically relevant, an assumption commonly adopted in the information elicitation literature [10]. The following lemma shows that the stochastic relevance requirement directly translates to a constraint on the error rates, that is, e 1,z + e 0,z = 1. Our goal is to design a scoring function ϕ : I × {0, 1} → R + that satisfies that there exists a strictly proper scoring rule S and a strictly increasing function f : R + → R + such that For instance, a simple such f to aim for can be an affine function that there exist two constants c 1 > 0, c 2 such that E y,z [ϕ(a i , z)] = c 1 · E y [S(a i , y)] + c 2 .
If our goal is achieved, we have preserved the same order of quantification of information as strictly proper scoring rules do, due to the monotonicity of f (·). Further with defining the strict properness for this setting with noisy ground truth: Strict properness with noisy ground truth: where the expectation is taken with respect to the agent's subjective belief Pr[z|s i ], the strict properness and a dominate strategy elicitation of ϕ(·) hold immediately because of the strict properness of S and the monotonicity of f (·).

Model of elicitation with peer reports
The second elicitation setting is close to the standard setting considered in the peer prediction literature.
Here the only source of information that the mechanism designer has access to is the agents' reports. There are multiple tasks in this setting. Each task k has a ground truth y(k) that is independently drawn according to prior P 0 and P 1 . We make a technical assumption that P 0 = P 1 . 4 We adopt the standard assumption requiring that each agent's signal s i (k) and the ground truth y(k) are stochastically relevant, i.e. e 1,i + e 0,i = 1. Moreover, we assume that signals are Bayesian informative of y(k), that is, Pr[y(k) = s|s i (k) = s] > Pr[y(k) = s], ∀i, s. In other words, the posterior probability for y(k) = s is greater than the prior probability of y(k) = s if an agent receives a signal s. This assumption has been adopted by [10,15], and has been shown by [29] to be equivalent to a constraint on the error rates: is Bayesian informative of y if and only if e 1,i + e 0,i < 1.
The case with e 1,i + e 0,i > 1 can be viewed as negative Bayesian informative as flipping one's observation will return a Bayesian informative signal. In this paper we focus only on settings where every agent's signal is Bayesian informative of y. But this assumption is to simplify presentation as our results hold for negative Bayesian informative agents, as well as for a mixed population.
Each task will be randomly assigned to three agents. Suppose agent i receives n i tasks. We do not require each agent to take on a large number of tasks. In particular, when the total number of agents is large, on average each agent only receives a small number of tasks (for instance, n i ≤ 2). We denote agent i's private information as I i (n), n = 1, ..., n i for the n i tasks. Suppose each agent i incurs the same error rates e 1,i , e 0,1 across all tasks assigned to him.
We use σ i : I → ∆(I) to denote agent i's reporting strategy, where ∆(I) is the simplex over the information space. Let a i (n; σ i ) ∼ σ i (I i (n)), n = 1, ..., n i denote agent i's realized reports following his strategy σ i . By this, we have also assumed that each agent follows the same reporting strategy for all tasks he is assigned. We'd like to emphasize that neither the homogeneous error rates nor the same reporting strategy assumption is very restrictive when n i is small, which is the case in our setting. Dealing with taskdependent strategies is a challenging (if not impossible) task as noted in the literature of multi-task peer prediction literature [15,30].
Denote by D −i the set of information collected from agents j = i who play some (unknown) strategies {σ j } j =i , i.e. D −i := {a j (n; σ j ), n = 1, ..., n j } j =i . For this part, our goal is to find a reference signal z D −i ∈ {0, 1} 5 for each a i (n; σ i ), as a function of D −i , and a scoring function ϕ D −i (a i (n; σ i ), z D −i ) : I × {0, 1} → R + , also as a function of D −i , such that when z D −i is informative, there exists a strictly proper scoring rule S : I × {0, 1} → R + and a strictly increasing function f : ∀i, ∀σ i , ∀a i (n; σ i ), and ∀n, and S is only required to be proper if z D −i is uninformative.
If the above holds, besides preserving the quantification of information, it is proper for agent i to truthfully reveal his private information regardless of other agents' strategies 6

. Then
Multi-task properness with peer reports: ∀i, ∀n, ∀σ j , j = i, and ∀a i (n; σ i ) and when z D −i is informative about y, it is strictly proper for agent i to truthfully reveal his private information: Multi-task strict properness with peer reports: and the expectation is taken with respect to the agent's subjective belief Pr[z D −i |I i (n)]. Note we haven't restricted how σ j 's are chosen by agents j = i. The above conditions need to hold for any strategy profile {σ j } j =i . Thus, for the Bayesian games that agents are playing, the above conditions require that for any strategy profile of other agents it is the best strategy for agent i to report his information truthfully. Thus, truthful reporting is a dominant strategy in this multi-task Bayesian game. We note that however the above conditions do not mean that truthful reporting is an ex-post dominant strategy after the reports of other agents are realized. In fact, in information elicitation settings (even with ground truth), it's impossibly to achieve truthfulness as an ex-post dominant strategy. (Once the outcome or reference report is known, it's always better to report the same rather than report one's information.) The reason that it's possible for us to achieve truthfulness in dominant strategy for the Bayesian game, but not the ex-post dominant strategy, is the assumption that each agent adopts the same strategy across all tasks. When it's clear in context, we'll use z as shorthand for z D −i and ϕ for ϕ D −i .
Knowledge of agents and the mechanism designer For signal elicitation, agents do not need to be aware of their own values of e 1,i and e 0,i , but they need to know that their own signals are Bayesian informative. For prediction elicitation, agents need to know their own values of e 1,i and e 0,i to form their predictions. The above knowledge does not need to be common knowledge among agents. This is because we seek to achieve dominant strategy truthfulness and hence agents are as if not playing a Bayesian game against each other. The mechanism designer knows the priors of distribution of ground truth P 0 , P 1 , i.e., we know among 100 images roughly 60 of them are of label 1, but not the realized y (which 60 are 1), nor the private information s i 's or p i 's. This assumption for now is for the ease of presentation. Later, as well as in the Appendix, we will show that we do not require the exact knowledge of P 0 , P 1 . Instead we only need to know an indicator function of 1(P 0 > 0.5). Further the designer does not know any of the biases of agents' information, e 1,i and e 0,i , a-priori.

Surrogate scoring rules
In this section we first introduce surrogate scoring rules (SSR) for the setting where the elicitation is done with noisy ground truth -we will name the scoring functions that meet our goals defined in Section 3.1 as surrogate scoring rules.
Definition 2 (Surrogate Scoring Rules). ϕ : I × {0, 1} → R + is a surrogate scoring rule if for some strictly proper scoring rule S : I × {0, 1} → R + and a strictly increasing function f : The above definition seeks a surrogate scoring function ϕ(·) that helps us remove the bias in z to return us a strictly proper score in expectation. The idea is borrowed from the machine learning literature on learning with noisy data [21,23,24,27,28]. SSR can be viewed as a particular class of proxy scoring rules [31]. But the approach of [31] to achieve properness is to plug in an unbiased proxy ground truth to a strictly proper scoring rule. SSR on the other hand directly work with biased proxy and the scoring function is designed to de-bias the noise. Easily we have the strict properness of SSR: We give an implementation of SSR, which we name as SSR alpha: where in above S can be any strictly proper scoring rule. Eqn. (4) is well defined due to the fact that z is informative and Lemma 2. From above we note that the knowledge of the error rates e 1,z , e 0,z is crucial for defining a SSR. The above scoring function is inspired by the work [23]. Intuitively speaking, the linear transform will ensure that in expectation, a i is scored against y: This can be proved fairly straightforwardly via spelling out the expectation. Interested readers are also referred to [23]. We would like to note that other surrogate loss functions designed for learning with noisy labels can also be leveraged to design SSR. With Theorem 4 we know that (SSR alpha) quantifies the quality of information just as the strictly proper scoring rule S does. Nonetheless, (SSR alpha) has a variance that is inversely proportional to 1 − e 1,z − e 0,z : Theorem 5. (SSR alpha) suffers the following variance:

Dominant truth serum
The results in the previous section are built upon the fact that there exists a reference signal for the ground truth and we know its error rates. In this section, we apply the idea of SSR to the peer prediction setting. A reasonable way to do so is to take agents' reports as the source for this noisy copy of the ground truth. Yet the mechanism designer cannot assume the knowledge of the noise in agents' reports.
We select the reference report z uniformly randomly from agents j = i. Denote by e r 1,i , e r 0,i the error rate of agent i's reported information a i s 7 . For the signal elicitation setting, for instance when agents truthfully report, i.e. a i (n; σ i ) = s i (n), we have e r 1,i = e 1,i , e r 0,i = e 0,i . For prediction elicitation, we can similarly define a signal from agent i (denoting as p r i (n; σ i )), via drawing a random sample according to his reported prediction: a s i (n; σ i ) ∼ Bernoulli(p r i (n; σ i )). In this case, we will still call such a a s i (n; σ i ) as agent i's "signal" but we should keep in mind this signal is randomly drawn from his reported prediction, instead of being deterministic. We similarly define e r 1,i , e r 0,i for a s i (n; σ i ). Then e 1,z and e 0,z can be characterized as follows: The next subsection will detail the steps towards estimating e 0,z , e 1,z . Note if we know the average error rates when agents truthfully report, the SSR already give us an equilibrium implementation of truthful elicitation -as in the equilibrium argument, every other agent will truthfully report.

Bias learning with matching
Our learning algorithm for inferring e 0,z , e 1,z relies on establishing three equations towards characterizing e 1,z , e 0,z . We will first show that the three equations, with knowing their true parameters, together will uniquely define e 1,z , e 0,z . Then we argue that with estimated and imperfect parameters from agents' reports, the solution from the perturbed set of equations will approximate the true values of e 1,z , e 0,z , with guaranteed accuracy.
(1) Posterior distribution: The first equation is based on the posterior distribution of 0/1 labels collected in D −i , denoting as P 0,−i := Pr[z = 0], P 1,−i := Pr[z = 1]. P 0,−i can be characterized as, via spelling out conditional expectation: (2) Matching between two signals: The second equation is derived from a second order statistics, namely the matching probability. Consider the following experiments: draw two signals randomly (for the same task, but from two agents) from the reference agents, denote them as z 1 , z 2 . Denote the matching-on-1 probability of the two signals as Pr[z 1 = 1, z 2 = 1] := q 2,−i . Further the matching probability can be written as a function of e 0,z , e 1,z : the quadratic terms are due to the fact that agents' reports are conditionally independent on ground truth: In practice, we won't obtain two identically distributed copies of answers for the same question with the same e 1,z , e 0,z . However when the number of agent is large enough, we will show that drawing two or three agents randomly from the population (without replacement) can approximate these e 1,z , e 0,z with small and diminishing errors (as a function of number of agents N ). This can factor into the errors in estimating q 2,−i .
(3) Matching among three signals: The third equation is obtained by going one order higher that, we check the matching-on-1 probability over three signals drawn randomly from (three) reference agents. Similarly as defined for Eqn. (7) matching between two signals, draw three independent signals (for the same task) from the reference agents, denote them as z 1 , z 2 , z 3 . Denote the matching-on-1 probability of the three signals as Pr[z 1 = z 2 = z 3 = 1] := q 3,−i . Then we also have that (details omitted) Theorem 6. (e 0,z , e 1,z ) is the unique pair of solution to Eqn. (6,7,8), when P 0 = P 1 .
This result implies that without ground truth data, knowing how frequently human agents reach consensus with each other will help us characterize their (average) subjective biases. Furthermore, releasing P 0 to be another unknown variable, we prove that we do not need to know the exact priors to really solve the equations: Theorem 7. (P 0 , e 0,z , e 1,z ) are uniquely identified using the three matching equations when we know the signal 1(P 0 > 0.5).
Further we show that the three equations are not only necessary but also sufficient: Theorem 8. The higher order (≥ 4) matching equations do not bring in additional information.

Statistical consistency results
All three parameters P 1,−i , q 2,−i , q 3,−i can be estimated from agents' reports, without the need of knowing any ground truth labels. To enable the estimation, our method relies on multiple tasks (therefore our mechanism is naturally a multi-task one). Among all assigned tasks, suppose there are K tasks that have been assigned to agents j = i -this is not hard to guarantee ∀i when we assign more than K tasks to the entire set of agents randomly.
The algorithm first estimates the following three quantities for each agent i:q 2,−i ,q 3,−i andP i 1,−i based on above information. Denote the three agents that are assigned task k = 1, ..., K as r 1 (k), r 2 (k), r 3 (k) = i, and their reports for task k asỹ r idx (k) (k), idx = 1, 2, 3. Then we estimate: We can then solve the system of equations (6,7,8) with these estimates to obtain estimated error rates e 1,z ,ẽ 0,z . We present the algorithm and a set of statistical consistency analysis for the case when we know P 0 . The ones for the case with unknown P 0 are similar but messier.
Nevertheless the Left side figure shows a nice decay in the errors of estimating e 0,z , e 1,z . The Right side figure shows the gap in expected score between truthful reporting and misreporting (denoted as "Reward gap") with the estimated e 0,z , e 1,z (for the same simulation setting). Indeed we show truthful reporting is better (positive gap, increasing with more accurate estimations of e 0,z , e 1,z ) even when majority of people are wrong (as e 0,z = 0.85, e 1,z = 0.83).

Information quantification and truth-telling is a dominant strategy
We define dominant truth serum (DTS) as follows: Mechanism 2 Dominant Truth Serum (DTS) 1. For each task agent i received and reported information, randomly select one of the rest two reference reports (recall each task is randomly assigned to three agents) as the reference z. 2. When z is uninformative thatẽ 1,z +ẽ 0,z = 1: -Score agent i zero regardless of his report.
Theorem 9 readily implies that DTS is asymptotically (in K, N ) preserving the information quantification as strictly proper scoring rules do and induces a dominant strategy for agent to report truthfully, when z is informative (proper otherwise). To see this, suppose bothẽ 1,z ,ẽ 0,z have converged to their true values, we have E[φ(a i (n; σ i ), z)] → E[S(a i (n; σ i ), y)]. Formally, Theorem 10. When z is informative, asymptotically the expected score of DTS equals to the score of its corresponding strictly proper scoring rule S, and it is a strictly multi-task dominant strategy for agent i to report truthfully. DTS induces a (weakly)-multi-task dominant strategy when z is uninformative. Remark 1. Several remarks follow. (1) We would like to emphasize again that both z and ϕ come from D −i : z will be decided by agents j = i's reports D −i . ϕ not only has z as input, but its definition also depends on e 1,z and e 0,z , which will be learned from D −i . (2) When making decisions on reporting, we show under our mechanisms agents can choose to be oblivious of how much error presents in others' reports. This removes the practical concern of implementing a particular Nash Equilibrium. (3) Another salient feature of our mechanism is that we have migrated the cognitive load for having prior knowledge from agents to the mechanism designer. Yet we do not assume the designer has direct knowledge neither; instead we will leverage the power of estimation from reported data to achieve our goal. Now we show the results in finite sample regime under noisy estimations. We first define the informative region. When e 0,z +e 1,z is arbitrarily close to 1, we have the difficulty in determining the number of samples needed for the learning process (to decide whether z is informative or not). With this in mind, we will modify our mechanism as follows: when |ẽ 1,z +ẽ 0,z − 1| ≤ κ (instead of setting it to be exactly 0) for some small positive constant κ, score agent i nothing, i.e., z is treated as being uninformative. κ can then help us quantify the number of K, N needed.
Theorem 11. When z is informative, set κ small enough and K, N large enough but finite, DTS returns a score that is (K, N ) close to the score of its corresponding strictly proper scoring rules, where (K, N ) = O N −1 + K −1 log K is a diminishing term in both K and N . Further with DTS, for each agent i, (1) for signal elicitation, it is a strictly multi-task dominant strategy to report s i (n), ∀n truthfully. (2) For prediction elicitation, it is a strictly multi-task dominant strategy to report p i (n), ∀n truthfully when S(·) is strictly concave in report a i (n; σ i ) and Lipschitz. (3) For prediction elicitation, it is an (K, N )-approximately strictly multi-task dominant strategy to report p i truthfully for any Lipschitz S(·).
Above we adopt the approximately multi-task dominant strategy definition as follows: truth-telling is -multi-task dominant strategy if E[ϕ(I i (n), z)] ≥ E[ϕ(a i (n; σ i ), z)] − , ∀a i (n; σ i ) = I i (n).

Experiments
Since we have made assumption on the homogeneous error rates across tasks for agents in proving the theoretical properties of DTS, we validate DTS using several datasets collected from human subjects. For each dataset, human participants report binary answers to each of the questions, such as "Is this art work's price higher than $30k? (y/n)". For several datasets, participants also report confidences (a continuous number in [0, 1]) in their answers. For evaluation purposes, we will convert this number into a probabilistic prediction. 10 For readers who are interested in the full details of these datasets, please refer to [32]. We summarize some basics in our full version draft. 1/Prior score is used to evaluate participants' binary answers and we use Brier scoring rule to evaluate probabilistic predictions. For each participant, we compute their true performance scores for each task they performed using ground truth answers and take an average. Then we compute the "surrogate" performance scores for each participant using DTS (without accessing the ground truth). To support our theory, we expect to see that the surrogate scores approximate the true scores, and reflect the differences in participants' true performances. In Fig. 2, participants are ordered by their true performance scores (in decreasing order). We do observe that the surrogate scores (plotted next to the true scores) approximate the true scores well, while the scores returned by peer prediction scores (Peer Truth Serum (PTS) [16] and Bayesian Truth Serum (BTS) [32] 11 ) 9 We thank the authors for sharing the data with us. 10 When this number is missing, we use another prediction from agent to make such a conversion. For our evaluation purpose, it doesn't really matter how this number is generated. 11 The comparison with BTS may not be completely fair as the scores are looking to be on different scales. But the BTS scores generally don't rank participants similarly as the rank according to the true scores. generally do not reflect participants' differences in true performance scores (see Fig. 2c). More results on other datasets can be found in our full version.

Human judgement datasets
Detailed empirical results are summarized in Table 1. Our error rate estimation algorithm works reasonably well for Studies 1, 2, and 4. Note this is done without assuming the knowledge of marginal priors P 0 , P 1 . The average errors (averaged across all participants) in approximating Brier scores are smaller as the scores are continuous, as compared to the 1/Prior scores. We highlighted in Study 4 that participants are more likely to be wrong (e 0,z > 0.5) for a certain label class; yet our estimation procedure was able to identify the high error rate.

Human forecasting dataset
We also carried out studies over a set of large scale human forecast data collected from the Hybrid Forecasting Competition organized by IARPA. In this dataset, we have in total > 1000 participants contributed predictions for over 100 geo-political events in a time span of 6 months.For instance, the participants were asked questions such as "Will event X happen before August 1st, 2018?" Each participant will give their prediction p ∈ [0, 1] for the event they chose to forecast on. For each participant, we compute their true Brier scores (B.S.) for each task (denoting as IFPs) they predicted using the ground truth answers. We then average them across events. We further compute the DTS scores for each participant without accessing the   Fig. 3, We observe that the DTS scores approximate the true scores (i.e, the dots plot approximately aligns with the line of y = x), and reflect the differences in participants' true performances.

Concluding remarks
We propose SSR and DTS to quantify the value of elicited information in IEWV settings, as strictly proper scoring rules do for the with verification setting, which complement the literature of strictly proper scoring rules by considering the setting when there is only access to a noisy copy of the ground truth. Our findings are both verified analytically and empirically.
Our work opens up the study of calibrating the value of information for the peer prediction setting. In the future, we hope to complement the current works via studying the cases with correlated signals, heterogeneous error rates, and machine learning aided reference report generation.

Additional materials
We fill in the missing proofs and figures, as well as solutions for the case when we have correlated signals.

A Proof of Lemma 2
Proof. We will not carry the index k for s i (k). Suppose not we will have  (1 − e 1,s ) · e 1,s , completing the proof.

C Proof of Theorem 4
Proof. Consider agent i, and the case that e 1,z + e 0,z < 1. The proof is straightforward following the "unbiasedness" property established for ϕ(·) in Lemma 4: The theorem follows immediately from the strictly properness of S. Now consider the case with e 1,z + e 0,z > 1. We cannot directly apply Lemma 4. Now let's define the "flip signal"ẑ of z:ẑ = 1 − z. Easy to see that and e 1,ẑ + e 0,ẑ < 1. The scoring function for agent i then becomes: Then we have where the last equality is due to the unbiasedness of ϕ(·) with respect toẑ (instead of z), and the fact that e 1,ẑ + e 0,ẑ < 1, so Lemma 4 can be applied. Again it is easy to see it would be agent i's best interest to tell the truth, due to the strict properness or truthfulness of S(·), finishing the proof.

D Proof of Theorem 6
Proof. First of all, there are at most two solutions from Eqn. (6) and (7), since the equations are at most order two. Further we prove the following properties of its solutions: Lemma 5. For the solutions to Eqn. (6) and (7), we have (1) when the reports are uninformative such that e 0,z + e 1,z = 1, the above equation returns exactly one solution satisfying that e 0,z + e 1,z = 1.
We shall shorthand the following: x 1 := e 0,z , x 2 := 1 − e 1,z . We now show the following fact: The above equation, together with equation (I), will uniquely identify the solution for x 1 , x 2 when P 0 = P 1 : This completes the proof.

E Proof of Theorem 7
Proof. Let's simplify the three matching equations in notation: where c 1 , c 2 , c 3 denote the first, second and third order matching statistics. Further denote by x := 1 − e 0,z , y := e 1,z , z := P 0 : From Eqn. (19) we know z(x − y) = c 1 − y ( * ). From Eqn. (20) and (21) we have respectively that z(x − y)(x + y) + y 2 = c 2 z(x − y)(x 2 + xy + y 2 ) + y 3 = c 3 Plug Eqn.( * ) into above two equations we know: (c 1 − y)(x 2 + xy + y 2 ) + y 3 = c 3 Denote x + y = a, xy = b, then a = b+c 2 c 1 from Eqn. (22). Note the above is well defined, as o.w. if c 1 = 0, we have to have x = y = 0 which leads to e 0,z + e 1,z = 1, which is a contradiction. Substitute (a, b) into Eqn. (23) we have Further a = b+c 2 . Now we show that c 2 = c 2 1 , so the above pair of solutions are well defined. Suppose not, we will have which contradicts that e 0,z + e 1,z = 1. Then from x + y = a, xy = b, there exists at most two solutions: Denote them as (x 1 , y 1 ) and (x 2 , y 2 ). By symmetry we know x 1 = y 2 , y 1 = x 2 . Further we have x 1 = y 1 and x 2 = y 2 , due to the fact again that e 0,z + e 1,z = 1. Since c 1 = z · x + (1 − z) · y, and z > 0, then it must be min{x, y} < c 1 < max{x, y}. Suppose x 1 < y 1 , then we will know that z = 0.5, contradicts the non-uniform prior assumption.

F Proof of Lemma 5
Proof. Eqn. (6) and (7) become equivalent with the following From the first equation (I) we know that x 1 = P 1,−i −P 1 ·x 2 P 0 . Then Further plug x 1 into the second equation (II) we have where ∆ := From above we have derived the following When e 0,z + e 1,z = 1, we know we must have ∆ = 0 so we land at a unique solution. When e 0,z + e 1,z = 1, we know that ∆ > 0 and we have two solutions -but both solutions satisfy that |e 0,z + e 1,z − 1| = ∆ P 0 , finishing the proof.

G Proof of Theorem 8
Proof. We follow the shorthand notations as in the proof of Theorem 7 (Section E). Consider the fourth equation: zx 3 + (1 − z)y 3 and zx 2 + (1 − z)y 2 are the second and third equation, while we know the first three equations already uniquely characterize x + y and xy, so the fourth equation is redundant. This sets up the induction basis. For any n > 5, we have By induction hypothesis we know zx n−1 + (1 − z)y n−1 and zx n−2 + (1 − z)y n−2 can both be written as functions of the first three equations, so is zx n + (1 − z)y n . Proved.

H Proof of Theorem 9
Proof. Estimation error due to heterogeneous agents: The first challenge lies in the fact that the higher order equations doesn't capture the true matching probability with heterogeneous workers. If we assign one task (to be labeled) to two different workers, due to the asymmetric error rate, the LHS of Eqn. (I) is not precise-drawing a worker without replacement leads to a different labeling accuracy, and will complicate the solution for the system of equations. We show that our estimation, though being ignoring above bias, will not affect our results by too much: denoting by e 1 the accuracy of first drawn sample (for checking the matching) when y = 1 (we can similarly argue for y = 0). Then we have E[e 1 ] = e 1,z . And the average accuracy of the second drawn sample e 2 (conditional on e 1 ) is given by Further the matching probability satisfies the following: Note both e 1,z and E[e 2 1 ] are no more than 1. Then That the estimated quantity is bounded away from the true parameter by at most Θ(1/N ). Similarly for the third order equation we have Estimation errors due to finite estimation samples: The second sources of errors come from the estimation errors ofq 2,−i ,q 3,−i andP 1,−i . We have the following lemma: Lemma 6. When there are K samples for estimatingq 2,−i ,q 3,−i andP 1,−i respectively (total budgeting 3K) and K ≥ log 6/δ 2 2 for any , δ > 0, we have with probability at least 1 − δ The above lemma can be easily established using Chernoff bound, and our arguments in estimating with heterogeneous agents above. We now bound the error in estimating p 0,z , p 1,z , under the ( , δ)-event proved in Lemma 6. First of all According to Lemma 7 of [33], which we reproduce as follows: We know the following facts First we prove that Let 3 + 2 N −2 ≤ P 0 P 1 κ 2 /2, we know that (using mean-value theorem/inequality) Together we proved that Similarly we are able to work out sensitivity analysis for e 1,z that

Summarizing and set
. With the error rates bounds for e 0,z , e 1,z , from Lemma 5.4 [29], which we reproduce in our contexts as follows: We then obtain a bound on estimating ϕ(·).

I Proof of Theorem 10
Proof. First for the case thatẽ 1,z +ẽ 0,z = 1, it is indifferent for agent i to truthfully report, or to misreport, or to randomize between the two strategies. Thus truth-telling is a weakly dominant strategy. Whenẽ 1,z +ẽ 0,z = 1, the multi-task dominant strategy argument follows from the strictly properness of (SSR alpha).

J Proof of Theorem 11
Proof. The proof applies to all assigned tasks n = 1, ..., n i so we will drop the dependency on index n. When it's clear in the context, we will also drop a i 's dependence on σ i and simply write the reported information as a i . Suppose |e 0,z +e 1,z −1| ≥ 2κ. Then when the estimation errors are small enough such that |ẽ 0,z −e 0,z |+ |ẽ 1,z − e 1,z | ≤ κ, we know that |ẽ 0,z +ẽ 1,z − 1| ≥ κ. Denote by ∆ ϕ = E[S(s i , y)] − max a i =s i E[S(a i , y)], the minimum gap between the scores for truthfully reporting and mis-reporting. We can easily show that ∆ ϕ > 0 for signal elicitation case, as we can take a i to be the three basis of mis-reporting: always reverting the observation, always reporting 1 and always reporting 0. Then with a noisy estimation of ϕ(·), we have (using Theorem 9) Above, we implicitly assumed the boundedness ofφ(·): notice with bounded scoring function S, indeed we know that maxφ i.e., the strict properness will preserve, under the noisy estimations. From above results we also observe that a larger ∆ ϕ will allow more noisy estimations. We thus can trade more payment with sample complexity, via designing the strictly proper scoring functions to increase ∆ ϕ . Now consider the prediction elicitation case. Suppose that S(p, y) is strictly concave w.r.t. p ∀y with parameter λ.φ(a i , y) = ϕ(a i , y) + (a i , y), where (a i , y) indicates the error term.
Noticing that using strictly concavity we have Due to the sample complexity results we know that with probability at least 1 − δ that y) is also Lipschitz w.r.t. p ∀y with parameter L. By Lipschitz conditions we know that with probability at least 1 − δ that (a i , y) is Lipschitz with parameters 2 1 L by composition property and With probability at most δ that i.e., (a i , y) is Lipschitz with parameters 2 κ L by composition property and Combining we have Therefore when 1 , δ are small enough such that no deviation is profitable.
K -dominant strategy argument follows naturally from above error term analysis. We will not repeat the details.

K Discussions
Below we make a couple of remarks, discuss possible caveats and their possible fix.

K.1 Weak dominance v.s. strong dominance
We have shown when we cannot learn an informative enough reference report from the rest of the agents, there is very little we can do regarding that. With this note, our deterministic payment strategy is only weakly dominant. We argue that if agents' reports can be modeled as being from certain distributions, the weak dominance case happens rarely.
Denote by X i =ẽ 1,i +ẽ 0,i , the weakly dominant case happens only when j =i X j = 1. Potentially there are infinitely many strategies that lead to this state. However, suppose agents' reporting strategies are drawn from a distribution defined over a continuous space (mixed strategy space), then we can easily argue that {ω : j =i X j (ω) = 1} (via taking j =i X j as a random variable) is a zero measure event.
Also we would like to point out that, as long as each agent believes that this uninformative state happens with probability < 1, DTS induces strong dominant strategy in truthful reporting, as a non-trivial mix between weak and strong dominance returns strong dominance.
Also, as a matter of fact, j =i X j = 1 corresponds to the case that the reference signal is stochastically irrelevant to the ground truth. Theoretically speaking, there is little one can do for eliciting private signals using such a reference signal, via a peer prediction method. In practice, one can use screening tasks to remove some workers to move away from this uninformative state.

K.2 Effort exertion
So far our discussion focuses on eliciting truthful reports, our results extend to the scenario for eliciting high quality data. For instance, when workers can choose to exert costly effort (consider a binary effort level case, and denote the cost as c for exerting efforts) to improve the quality of their answers, and once the reference signal is informative (either positively or negatively), we can scale up the scoring functions to cover the cost c, as similarly done in [10], to establish the dominance of reporting a high-effort signal.

K.3 Heterogeneous error rates across tasks
In order to estimate error rates correctly, we needed to make the assumption that human workers have homogeneous error rates across multiple tasks. We now discuss the applicability of our method in light of the heterogeneity issue. Before starting, we would like to emphasize that in practice, for each set of experiments we can choose to group tasks according to their types (e.g., image labeling, solving puzzles, objects recognition), and run our mechanism over each group separately. For this setting, we can assume the homogeneity more comfortably.
Nonetheless when it is not quite possible to group tasks together, suppose that each human agent's error rates are also task contingent in that for each possible task x with y ∼ P := {P 0 , P 1 } we have e 0,i (y), e 1,i (y). Redefine the error rate of the reference answer as follows: Again e 1,z , e 0,z captures the error rate of the reference answer. With knowing e 1,z , e 0,z , the dominant strategy argument in Theorem 4 holds. So far so good. However it becomes less clear in how to estimate e 1,z , e 0,z . If we follow the idea in Section 5.1, the first order equation holds as before i.e., P 0 e 0,z + P 1 (1 − e 1,z ) = P 1,−i . However the higher order matching statistics appear to be very different.
For example, the second order matching probability between two agents on label 1 becomes Instead we would like a equation as a function of in order to identify the true error rates on average. To get around of this issue, we can adopt the following approximation for (e r ) 2 0,j (x) using Taylor expansion: (e r 0,j (x)) 2 ≈ e 2 0,z + 2e 0,z (e r 0,j (x) − e 0,z ) = 2e 0,z · e r 0,j (x) − e 2 0,z .
Then we claim that From above, we see we are able to recover the matching equations defined in our earlier solution for task homogeneous cases; and it can be viewed as a linear approximation for this task contingent case.

K.4 A Machine Learning method
To echo recent works on using machine learning techniques [29] to learn a machine learning model to generate a reference answer, instead soliciting from other agents, we show this idea is also ready to be plugged into our SSR solution framework. Consider the current task that needs to be elicited and denote its feature vector as x ∈ R d . Suppose we have learned (following the results in [29], such a classifier is learnable purely from agent's reported noisy data) a good classifierf * (x) for predicting its true outcome.
Replace the reference answer z withf * (x) and plug in its error rates. The rest of job is to reason about e 1,z , e 0,z (forf * (x)). There are possibly many different ways to do so. Suppose that each task is associated with a corresponding feature vector. Consider the current task that needs to be elicited and denote its feature vector as x ∈ R d . Suppose we have learned (following the results in [29], such a classifier is learnable purely from agent's reported noisy data) a good classifierf * (x) for predicting its true outcome. Our ML-aided SSR simply works in the following ways: replacing the reference answer z withf * (x) and plug in its error rates. The rest of job is to reason about e 1,z , e 0,z (forf * (x)). There are possibly many different ways of doing so. We demonstrate its possibility when there is again no ground truth label being available with the following simple estimation procedure: • Assign K tasks x 1 , x 2 , ..., x K to three randomly drawn peers, denote them as z ref Then note the following fact: Together with With above argument, we see that we will be able to achieve dominant strategy scorings without even the need of reassigning all tasks.

Additional experiment results
We add experiment results for all other datasets. Since BTS scores are on a much smaller scale compared to other scores, we scale up BTS scores by a constant, to make the comparison more clear.