{'fraction_non_alphanumeric': 0.028132753763781878, 'fraction_numerical': 0.017163218476577802, 'mean_word_length': 4.851326274424189, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 1, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Assessing the Trustworthiness of Crowdsourced Rainfall Networks: A Reputation System Approach

High resolution and accurate rainfall information is essential to modeling and predicting hydrological processes. Crowdsourced personal weather stations (PWSs) have become increasingly popular in recent years and can provide dense spatial and temporal resolution in rainfall estimates. However, their usefulness could be limited due to less trust in crowdsourced data compared to traditional data sources. Using crowdsourced PWSs data without a robust evaluation of its trustworthiness can result in inaccurate rainfall estimates as PWSs are installed and maintained by non‐experts. In this study, we advance the Reputation System for Crowdsourced Rainfall Networks (RSCRN) to bridge this trust gap by assigning dynamic trust scores to PWSs. Based on rainfall data collected from 18 PWSs in two dense clusters in Houston, Texas, USA as a case study, we found that using RSCRN‐derived trust scores can increase the accuracy of 15‐min PWS rainfall estimates when compared to rainfall observations recorded at the city's high‐fidelity rainfall stations. Overall, RSCRN rainfall estimates improved for 77% (48 out of 62) of the analyzed storm events, with a median root‐mean‐square error (RMSE) improvement of 27.3%. Compared to an existing PWS quality control method, results showed that RSCRN improved rainfall estimates for 71% of the storm events (44 out of 62), with a median RMSE improvement of 18.7%. Using RSCRN‐derived trust scores can make the rapidly growing network of PWSs a more useful resource for hydrologic applications, greatly improving knowledge of rainfall patterns in areas with dense PWSs.


Introduction
Flooding is becoming commonplace in cities and communities worldwide, causing severe damage and loss of property (Salman & Li, 2018;Wilby & Keenan, 2012). As a result of climate change, rainfall extremes are expected to become more intense and highly heterogeneous (Ohba & Sugimoto, 2019;Sharma et al., 2018). Floods triggered by these increased storms often exhibit large variability both in space and time, especially in urban areas with a large portion of impervious surface (Cristiano et al., 2017;Quinn et al., 2019). Although recent advances in computational power and modeling approaches have made it possible to accurately model flooding at increasingly high resolution (Mosavi et al., 2018;Saksena et al., 2019;Savage et al., 2016;Shen et al., 2019;Zahura et al., 2020), these models require measured rainfall observations as input at high spatial and temporal resolutions. However, the current resolution of observations through traditional rainfall networks is typically insufficient, or even unavailable, for certain flood-prone regions (Cristiano et al., 2017;Sadler et al., 2018;Zhu et al., 2018).
Traditionally, in situ rainfall observations are obtained from gauges managed by federal or municipal agencies. These rain gauges, which we refer to as high-fidelity rainfall stations in this study, provide accurate measurements as they are installed and maintained by experts, but they are limited in coverage (Overeem et al., 2013;Villarini et al., 2008). An alternative to rain gauges is the use of weather radars. However, radar rainfall is derived indirectly from radar reflectivity observed at certain heights in the atmosphere, which may not accurately represent rainfall at the ground level (Smith et al., 1996) and requires calibration with ground gauges (Krajewski & Smith, 2002). Recent improvements in dual-polarization technologies in weather radar address some of these limitations. However, further improvement of rainfall estimation using dual-polarization radars is needed. For example, range-dependent sampling errors and uncertainties in identifying hydrometeor types with radar measurements may introduce larger bias in rainfall estimates, which cannot be easily corrected with ground gauges (Cunha et al., 2015).
The increased popularity of PWSs can be attributed to the availability of the crowdsourced networks that allow anyone to act as a data contributor. Such openness, however, introduces challenges in assuring accurate data Chapman et al., 2017;Muller et al., 2015;Meier et al., 2017). Crowdsourced networks are typically lightly controlled networks with few constraints and only basic quality-control processes. As a result, people have higher levels of confidence in data collected from high-fidelity rainfall stations, as there are fewer sources of error in their observations compared to crowdsourced data (Cox, 2011;Hunter et al., 2013). Like high-fidelity rainfall stations, PWSs can experience device errors, but they can also suffer from compromised setups, lack of routine maintenance, and other sources of error that are less common in high-fidelity stations (de Vos et al., 2017;Meier et al., 2017;Muller et al., 2015). For example, improper installation of PWSs, such as siting the station under a tree canopy or next to a building, can lead to consistently inaccurate readings. Likewise, the owner of the PWS might not routinely maintain and calibrate the device, which could lead to sensor drift and faulty observations. Beyond these cases, it is also possible in open crowdsourced networks in which people might deliberately manipulate data to produce misleading evidence (Huang et al., 2014;Sanchez et al., 2018). Therefore, a method to evaluate the trustworthiness of crowdsourced PWSs is needed before this rich and growing dataset can be used with confidence in hydrologic studies and decision making.
One way to address the problems with crowdsourced PWS data would be to adopt quality control and quality assurance (QA/QC) methods to detect, flag, or remove doubtful and erroneous data based on certain rules and thresholds (Båserud et al., 2020;Blenkinsop et al., 2017;de Vos et al., 2019;Estévez et al., 2011;Fiebrich et al., 2010;Niu et al., 2021). If other data from more trusted sources is available, then another method would be to evaluate the quality of crowdsourced rainfall data by direct comparison with them (de Vos et al., 2017;Muller et al., 2015). Existing methods, however, may not adequately address the needs of crowdsourced weather data, and specifically, rainfall observations. QA/QC methods designed for high-fidelity stations tend to focus on outlier detection that presumes a certain source of error (sensor malfunction) and may be less able to detect other sources of error (poor sensor siting or installation). For example, the Weather Underground designates a PWS as a "Gold Star Weather Station" if it passes basic quality control criteria such as data validity, and a sensor failure checks over the previous five days (The Weather Channel, 2018). Direct comparison with trusted data sources presumes that such data is available, but PWSs have reached a density of observation in space and time that cannot be matched with other, more trusted, measurement methods. There is a need and opportunity, therefore, for innovative methods for assessing the data generated by PWSs at scale so that trustworthy stations can be used 10.1029/2021WR029721 3 of 25 more confidently in decision making and, just as importantly, untrustworthy stations can be reported to owners with suggestions for improving data quality so that the overall observation network reaches its full potential.
In this study, we explore the use of reputations systems as an approach for measuring the trustworthiness of rainfall observations from PWSs. Reputation systems are commonly used to build trust between participants and foster good behavior in online crowdsourced systems (Jøsang et al., 2007). For example, online markets such as eBay and Amazon use reputation systems to enhance users' buying and selling experiences. Such systems aggregate sellers' past behavior and represent it as a trustworthiness rating for buyers to rely on (Resnick et al., 2000). Reputation systems have also been used for citizen science and crowdsdourced data. Yang et al. (2013) designed a reputation system framework for enhancing the data reliability of citizen science environmental acoustic data. Silvertown et al. (2015) used a reputation system to motivate and reward participants of a crowdsourced species-identification website that improved the accuracy of species determinations. Huang et al. (2014) proposed a reputation system framework using the Gompertz function to compute device reputation scores based on the trustworthiness of the contributed data in participatory sensing applications. However, the use of reputation systems for crowdsourced PWS networks has not been widely adopted.
In our previous work, we presented an initial version of a system called the Reputation System for Crowsourced Rainfall Network (RSCRN)  to assign trust scores to PWSs. In this paper, we significantly enhance RSCRN and evaluate the method for storm events using Houston, Texas, as a case study. The research questions guiding this work are (a) How can we systematically evaluate the trustworthiness of crowdsourced PWSs? and (b) To what extent could a reputation system approach improve rainfall estimates derived from PWSs?
The remainder of the study is organized as follows. Section 2 describes details of the RSCRN algorithm and methods to evaluate the RSCRN for storm events. Section 3 provides a description of the study area and data used in the study, as well as a storm events selection process. The results and discussion of this study are presented in Sections 4 and 5, followed by conclusions in Section 6.

Data Preparation
The RSCRN method begins with a crowdsourced rainfall network in a specific region having N PWSs. Given an analysis period of interest (say X time steps), the rainfall observations from these N PWSs can be collected into a matrix P where p i,j is a rainfall observation measured at time step i and PWS j. This matrix P, with X rows for the rainfall observations and N columns of PWSs, will be used as the input for RSCRN.

RSCRN Algorithm
The RSCRN algorithm consists of three steps: Cluster, Consensus, and Score (Algorithm 1). The objective of RSCRN is to evaluate the trustworthiness of PWSs based on their consensus with rainfall measured at neighboring stations. The Cluster step is to find clusters of neighboring PWSs. Next, the Consensus step is used to identify PWSs with rainfall observations that deviate from a cluster's consensus. Finally, the Score step uses the degree of deviation from consensus to assign a new trust score to each PWS on a given time step that represents the trustworthiness of that PWS. Further detail for each step in the algorithm follows (Algorithm 1).
Python codes for the RSCRN algorithm and datasets used in this study are available from Hydroshare (https:// www.hydroshare.org/resource/cf7796cdeace42818dbbd7f95f8e1872/). For the Weather Underground, the P matrix can be populated for a region using their Application Programming Interface (API). Example code for this process is also provided as a resource in Hydroshare with the same link as above. This code requires a Weather Underground key to use the API, which at the time of this writing can be obtained by connecting a PWS to the Weather Underground platform.

Cluster
Different methods can be used to define PWS clusters in RSCRN. In earlier works, we explored methods for clustering including k-means where clusters are identified not only based on geographic position, but also on other factors including elevation . In this work, we take a simpler approach of defining clusters based on the geographic proximity of stations. First, we compute the number of PWSs within a buffered distance of a given PWS, p j . Second, PWSs that have (a) at least four of PWSs actively reporting rainfall observations during the analysis period of interest, and (b) at least one high-fidelity rainfall station within 2 km that can be used as a cluster for the analysis. For other applications of this method, the buffer distance may differ and should be selected based on the available PWSs within a region, and any local knowledge of rainfall autocorrection distances. Typically, distances of 4 kilometers or less should be selected (Sadler et al., 2017). Starting from the input matrix P, the resulting clustered matrices are denoted by , where k is the kth cluster, and M k is the number of PWSs in the kth cluster. These matrices will be the input for the consensus step.

Consensus
The input to the consensus step is clustered sub-datasets , where each sub-dataset contains rainfall observations from PWSs that fall within the same cluster. Assuming the kth clustered sub-dataset has m PWSs, this sub-dataset will be a matrix where p i,j represents the rainfall observation of PWS j within the cluster k measured at time i. For the clustered sub-datasets (k = 1, 2, …, K), the consensus step computes a cooperative metric (denoted as C i,j , which has the same dimension as ) based on the rainfall observations for each time-step (i = 1, 2, …, X) and each PWS (j = 1, 2, …, m).
We used the robust averaging algorithm (Chou et al., 2013) to estimate a cluster's consensus. We selected this method for its effectiveness and efficiency in similar applications for wireless sensor networks and participatory sensing (Ganeriwal et al., 2008;Huang et al., 2014). Robust averaging is a type of weighted average method that is less affected by values that deviate from the average. For each time step i, this iterative algorithm works as follows: 1. First, assign an initial (uniform) weight to every PWS j at iteration l = 1 where m is the number of PWSs in the clustered sub-dataset .
2. Next, compute the robust average , such that where p i,j is the rainfall observation of PWS j for time step i.
3. Next, compute the squared difference of PWS j's rainfall observation p i,j from the robust average 4. Finally, compute the new robust weight at iteration l + 1 The algorithm continues iterating until the convergence | − +1 | < is achieved, i.e., the robust weights converge to a value with difference less than ν. Note that the ϵ is a small positive constant that is set to 0.1, determined by trial and error based on the convergence of the algorithm (Chou et al., 2013).
The cooperative metric is then defined as where and σ(W i ) are the average and standard deviation, respectively, of the ith row of the robust weight matrix. This metric represents the level of deviation of the final robust weight from the initial weight. A positive cooperative metric indicates agreement with the consensus (robust average) within the cluster, while a negative cooperative metric represents disagreement with the consensus. The resulting cooperative metric is used as the input for score step.
We further extended the algorithm to accommodate two data exception cases: (a) all zero observations and (b) missing observations in certain time steps. In a clustered sub-dataset, the first case occurs when all rainfall observations are zero on a given time step. In this case, cooperative metrics will be invalid because the standard deviation of the robust weight matrix is zero. Therefore, the cooperative metric of every PWS in this case is set to zero. The second case occurs when PWSs have intermittent missing observations. In this case, for those time steps where a PWS has missing observations, this particular PWS is excluded from the robust average calculation, and the cooperative metric of this PWS will be set to zero. Additionally, if there are too many PWSs with missing observations resulting in a low number of active PWSs reporting data on a time step, the cooperative metrics of all PWSs on that time step will also be set to zero, because the consensus computed from the robust average algorithm may be unreliable. The definition of this low number can be determined based on the data availability during the analysis period.

Score
As described in Section 2.2.2, the cooperative metric can be interpreted as a measure of the PWS deviation from the robust average for each time step. To evaluate the trustworthiness of the PWSs, this step assumes neutral initial trust scores for every PWS without knowledge of any past behaviors (rainfall observations, in this case), and integrates this cooperative metric to update the trust score for every PWS for each time step.
We used the beta reputation system (Josang & Ismail, 2002) for its advantages of simplicity, flexibility, and ability to counter most arbitrary device faults in wireless sensor networks (Ganeriwal et al., 2008). The beta reputation system uses a statistical approach to provide a mathematical basis for trust management. The idea is that the trust score, which is computed based on the beta probability density function (PDF), is gradually updated as new observations are made available. The beta PDF is a continuous family of distribution functions indexed by two parameters: α and β. It is denoted by beta(p|α, β) and can be expressed using the gamma function Γ as where 0 <= p <= 1, α and β > 0. The expectation value of the beta distribution is given by where 0 < E(p) < 1.
In each clustered sub-dataset, the prior distribution is assumed to be a uniform beta PDF with α 1 = 1, β 1 = 1, and E(p) 1 = 0.5 for every PWS at time step i = 1 before any data are collected. This can be interpreted as the neutral trust for these PWSs, which indicates that the relative frequency of reporting trustworthy or untrustworthy observations is equal. After observing new data, the posterior distribution will be the beta PDF with updated α and β parameters. In RSCRN, these parameters are updated using the cooperative metrics C i,j computed from the consensus step as There are four possible outcomes for updating the α and β parameters: (a) A positive outcome is defined if the cooperative metric is greater than zero. In a positive outcome, the alpha parameter increases by the value of the cooperative metric. (b) A negative outcome is defined if the cooperative metric is less than −1, which implies significant deviation from the consensus because the final robust weight is more than one standard deviation below the average weight. In a negative outcome, the beta parameter increases by the absolute value of the cooperative metric. (c) A zero cooperative metric outcome indicates either all observations at the time step were zero, or there is a missing observation from a single PWS. In this case, both alpha and beta parameters are held constant with the previous time step values. (d) In a minor negative outcome-which we define as when the cooperative metrics is less than zero but greater than −1-both alpha and beta parameters will also be held constant with the previous time step values because the deviation from the consensus is insignificant. In addition, to focus the evaluation on time steps when rain is reported, the algorithm is set to update trust scores only on time steps when at least one PWS in the cluster is reporting more than one tick (0.25 mm) of rainfall.
A forgetting factor λ is introduced in Equation 8 to avoid the trust score being overly weighted on past information. The λ parameter, which ranges from 0 to 1, is used to give old information less weight than more recent information. A forgetting factor of 1.0 indicates no forgetting at all, whereas a forgetting factor of 0 indicates forgetting all past information except for the previous time step.
Given the updated alpha and beta parameters by the cooperative metrics, the expected value of the posterior beta PDF becomes Finally, the trust score T i,j is computed by re-scaling the expectation value to be between 0 and 10 for each PWS j at time step i

Comparison With a PWS Quality Control Method
The performance of the RSCRN approach is evaluated against a quality control method recently proposed for PWSs (de Vos et al., 2019). This quality control approach (hereinafter referred as PWS QC method) consists of four major filters to flag PWS rainfall observations. These filters are (a) a high influx (HI) filter to capture PWSs with observations much higher than neighboring stations, (b) a faulty zero (FZ) filter to identify erroneous zeros, (c) a station outlier (SO) filter to flag PWSs with low correlation of rainfall time series with neighboring stations, and (d) a bias correction (BC) filter to compensate for a possible bias in observations compared to the rainfall observations from neighboring stations. Using the same clustered sub-datasets as input for the PWS QC method, individual PWS observations were flagged with HI flags, FZ flags and SO flags, and these flags were used to compare to RSCRN trust scores. The BC filter was later applied in the validation step as described in the next section.

Validation Using High-Fidelity Rainfall Stations on Storm Events
Using a binary trust score threshold, PWSs were classified as trustworthy or untrustworthy PWSs for storm events with durations of X time steps that begin on time step T 1 and end on time step T 2 . The trust score thresholds of the each PWS are defined as follows: Trustworthy PWS: Untrustworthy PWS: where γ is the threshold value that ranges from 0 to 10.
Using γ = 5.0 as an example, trustworthy PWSs are stations that received average trust scores higher than 5.0 during this storm event. Generally, these PWSs are more likely to report trustworthy data during this storm event because they have been consistently contributing observations that agreed with the consensus from neighboring PWSs before the storm event. On the other hand, untrustworthy PWSs are PWSs that received average trust scores lower than 5.0, which indicates these PWSs have been reporting observations that disagreed with the consensus from neighboring PWSs. These PWSs are, therefore, less likely to report trustworthy data during the storm event.
To validate whether using a trust score threshold method can improve rainfall estimates from the crowdsourced rainfall network, we compute the root-mean-square error (RMSE) of the PWS rainfall observation with the nearest high-fidelity rainfall station for the storm event as where c i is the rainfall time series of the PWS, h i is the rainfall time series of the high-fidelity rainfall station, and X is the duration of the storm event. Consider a cluster with M PWSs: the average RMSE of all PWSs in the cluster (denoted as R all ) becomes where RMSE j is the RMSE of the jth PWS in the cluster. This R all is used to benchmark the improvement made from the RSCRN and PWS QC methods.
Assuming that the RSCRN trust score threshold revealed that in these M PWSs there are U trustworthy PWSs that received trust scores above the threshold, the average RMSE of trustworthy PWSs in the cluster (denoted as R RSCRN ) can be computed as The RMSE of trustworthy PWSs is further compared with the RMSE of QC PWS. In the comparison, the rainfall time series c i used in Equation 13 will be replaced with a bias corrected time series ′ resulting from the BC filter of the PWS method (de Vos et al., 2019). The RMSE (denoted as RMSE′) used in the PWS QC method will then become In order to compare the RSCRN and the PWS QC method, rather than using the flags resulting from the PWS QC method for individual rainfall measurements, the flags were translated into a binary decision of either at least one flag or no flags. Assuming there are V unflagged PWSs (stations without any flags filtered by the PWS QC method) during a storm event, the average RMSE of QC rainfall estimates (denoted as R QC ) can be be computed as Lower RMSE values indicate agreement with the high-fidelity rainfall station observations. The comparison of R all , R RSCRN , and R QC can then be used to determine the improvements of rainfall estimates made by each method in providing accurate rainfall estimation from a network of PWSs.

Study Area
To demonstrate and evaluate RSCRN, we focus on PWSs in Houston, Texas, as a case study. The City of Houston is in a sub-tropical climate with average annual rainfall of 1,250 mm. Flooding has been a recurring issue in Houston because of urbanization and the increase in frequency and intensity of severe storms . The growing adoption of PWSs in Houston in recent years significantly increases ground gauge rainfall networks coverage. Extracting trustworthy rainfall information from the PWSs could potentially supply denser point rainfall time series and, thus, improve the knowledge of rainfall patterns to better model and control flooding.

High-Fidelity Rainfall Network
In Houston, the Harris County Flood Control District (HCFCD) manages a rainfall monitoring network of 174 rainfall stations, with temporal resolutions up to 5 min, that can be used as the ground truth of the rainfall observation to evaluate RSCRN ( Figure 2). Additional information about the network as well as QA/QC reports can be accessed from the HCFCD website (https://www.harriscountyfws.org/).

Crowdsourced Rainfall Network
The crowdsourced rainfall network used in this study consists of PWSs available through the Weather Underground. We accessed the data through the API provided by the Weather Underground. The PWS observation sampling interval varies from station to station. Most of the sampling intervals are about 5-10 min per observation. Based on the available PWSs in the Houston area queried from the Weather Underground API, there were 99 PWSs in January 2016 and 382 PWSs just a few years later in April 2019.

PWS Cluster
In the cluster step, the buffer distance was set to 2 km so that the identified clusters satisfy the criteria described in Section 2.2.1. Two clusters with the most active PWSs available at the beginning of the analysis period were used to evaluate RSCRN (see Figure 2). The first cluster consists of eight PWSs and is located in southwestern Houston, Texas. The second cluster consists of 10 PWSs and is located northwest of downtown Houston. The inter-distance of PWSs in both clusters is less than 3 km ( Figure 3). Table 1 shows the available metadata from the Weather Underground API. Each PWS has different start times, which is when the station joined the Weather Underground and started reporting data to its database. Among these PWSs are three major PWS brands: Ambient Weather, AcuRite, and Davis Instrument.
The 15-min rainfall time series from the 18 PWSs in the two clustered sub-datasets for the analysis period from January 1, 2017 to March 28, 2019 were input to the RSCRN. In this study, the minimum number of valid rainfall observations on a time step for computing the cooperative metrics was set to 5, given that there are 5-7 PWSs actively reporting rainfall within a cluster for the majority of the time steps during the analysis period. The forgetting factor λ was set to 0.95, which retains approximately 20% of the prior knowledge that is more than 25 time steps (6 hr) old. This is to ensure that the trust score computed by RSCRN will not be overweighted by past observations so that it is able to accommodate temporary behavioral changes, especially during storm events. The sensitivity to this forgetting factor is further explored later in the paper. For the PWS QC method, the neighboring stations for each PWSs were set to all other PWSs in the cluster identified by the RSCRN. Several parameter choices were evaluated, and the best one was chosen based on the data availability and rainfall characteristics of the collected PWS data.

Storm Events Selection
In this study, a storm event is defined as accumulated rainfall greater than 25.4 mm within a 12-hr rolling window. Rainfall time series from the high fidelity rainfall network (HCFCD rainfall stations 445 and 560) were used to identify storm events for cluster 1 and cluster 2, respectively. As shown in Tables 2 and 3, 33 and 29 storm events with various rainfall statistics that occurred in what we refer to as winter (November to March) and summer (April to October) seasons during the analysis period (January 1, 2017 to March 28, 2019) were identified. In these storm events, duration ranged from 0.5 to 10.8 hr, maximum rainfall intensity from 20.3 to 113.8 mm/hr, and total rainfall from 25.4 to 164.6 mm.  Figure 4 shows the resulting trust scores based on the RSCRN for an example storm event. In this example, the majority of rainfall observations of the target PWS (KTXHOUST327) matched well with the robust average computed from neighboring stations in its cluster. At the beginning of the storm (marked with circle 1 in Figure 4), the trust scores remained unchanged because the RSCRN updates trust scores only when at least one PWS in the cluster reports rainfall greater than 0.25 mm. Similarly, in the middle and end of the storm (also marked with circle 1), the trust scores remain constant because all reporting rainfall is lower than 0.25 mm. The PWSs began to observe heavier rainfall starting at 02:45. As can be seen in Figure 4, the rainfall time series during the following time interval (marked with circle 2) agreed with the robust average. Therefore, the PWS received several positive outcomes (marked with the black dot on the Alpha axis), and the trust score steadily increased. There were two time steps when the PWS received negative outcomes (marked with circle 3 and black dots on the Beta axis) because its observations disagreed with the consensus, and the trust score decreased accordingly. Note that the change of trust score with regard to the same positive or negative outcomes was higher when using a smaller forgetting factor, because less prior knowledge was retained, and thus the trust score change was more sensitive. As expected, the PWS QC method did not identify any flags for the PWS during this storm event. Thus, both approaches agree this is a trustworthy PWS.  Figure 5 shows examples of when trust scores decrease for the majority of the time steps during a storm event for two untrustworthy PWSs. In the first example, the rainfall time series of the target PWS (KTXHOUST452) frequently deviated from the robust average. Although this PWS captured some of the peak values of the storm, there were several time steps between those peaks where rainfall observations significantly deviated from the consensus of the neighboring PWSs. For example, the consensus of rainfall observations among the neighboring stations was showing that it had been raining heavily between the time interval 03:00 to 06:00. However, this PWS was either reporting zero rainfall or underreporting rainfall, which resulted in receiving many negative outcomes (black dots on the Beta axis). Therefore, the trust score decreased and remained low for the entire storm. that this station is likely to be untrustworthy. In the second example, the rainfall observation from the target PWS (KTXHOUST1903) was underreporting (00:00 to 00:30) and reporting zero value rainfall while the neighboring stations showed strong consensus of a certain rainfall magnitude. This station also overreported rainfall at 03:30 while the consensus of the neighboring stations showed that the storm had stopped. Using the PWS QC method, this PWS was first flagged with FZ flags for several intervals, followed by an HI flag where this PWS was reporting 49.27 mm while other neighboring stations all reported zero. Thus, both approaches agree these are untrustworthy PWSs.

PWS Trustworthiness Assessment
The RSCRN trust score evolution over all analyzed storm events is shown in Figure 6. Note that the trust scores of each PWS were computed for every time step during the analysis period (1/1/2017 00:00 to 03/28/2019 23:45), and the trust scores for the analyzed storm events were extracted to assess the trustworthiness of a PWS during a particular storm event. In this figure, each dot represents the average trust score for a storm event. The trust score evolution shows that some PWSs were assigned high trust scores throughout the analyzed storm events  (e.g., KTXHOUST327 in cluster 1 and KTXHOUST805 in cluster 2), while other PWSs consistently received low trust scores (e.g., KTXHOUST617 in cluster 1 and KTXHOUST452 in cluster 2). However, some PWSs had trust scores that fluctuated over time, which indicates that perhaps these stations had state changes over the analysis period. Table 4 shows the overall assessment of PWS trustworthiness for the analyzed storm events. Based on the results for cluster 1, KTXHOUST617 was the least trustworthy PWS. If we assume 4.0 as the trust score threshold γ, of the 23 active storm events for which this PWS reported valid rainfall observations, 18 (78%) were classified as untrustworthy. If we use a more restrictive trust score threshold γ = 5.0, 20 (87%) storm events were classified as untrustworthy. KTXHOUST281 was the second least trustworthy PWS in this cluster, as its trust score fluctuated between 4.0 and 6.0, and eventually dropped below 2.5. Of the 31 active storm events this PWS actively reported, 10 (32%) were classified as untrustworthy with trust score threshold γ = 4.0. Notably, as shown in Figure 6, KTXHOUST1903 initially received high trust scores, but dropped below 5.0 during several storm events. However, its trust score was restored to above 5.0 after storm event 20170715, and remained mostly trustworthy for the rest of the time. Other PWSs, such as KTXHOUST1971, KTXHOUST327, and KTXHOUST355, received relatively higher trust scores and were classified as trustworthy for most of the storm events ( Figure 6). In cluster 2, KTXHOUST452 and KTXHOUST240 were the least trustworthy PWSs with an average trust score less than the threshold γ = 4.0 for 83% and 61% of the storm events, respectively. KTXHOUST443, with 29% of the storm events evaluated as untrustworthy received a high trust score at the beginning of the analysis period but decreased over time and eventually dropped below 2.5. Other PWSs were mostly trustworthy during the storm events based on the average trust scores they received.

Comparison With PWS Quality Control Method
A comparison with PWS QC method (Table 4) shows that for each PWS, the number of untrustworthy events (defined as storm events that average trust scores below a threshold) identified by RSCRN generally agreed with the number of flagged events (defined as storm events that had at least one observation flagged by the PWS QC method) for the analyzed storm events. In cluster 1, PWSs with a large number of untrustworthy events were  also frequently flagged by the PWS QC method, whereas PWSs that were assigned higher trust scores usually received fewer or no flagged events. In cluster 2, PWSs with a higher percentage of untrustworthy events also received several flags from the PWS QC method. Using a different trust score threshold for classifying PWSs, the comparison showed that the number of untrustworthy events agreed most with the flagged events from the PWS QC method when using γ = 4.0.
As most of the agreements between the RSCRN and PWS QC method were for high influx and faulty zero flags ( Figure 5), there were cases where RSCRN identified additional untrustworthy behavior, whereas there were no flags determined by the PWS QC method. Using the storm event January 20, 2017, as an example (Figure 7), the rainfall observed from this PWS (KTXHOUST281) received several negative outcomes from the RSCRN. At 17:30 and 18:15, the reported rainfall was 1.02 and 4.06 mm, while the robust average computed from the neighboring PWSs were 5.06 and 0.82 mm, respectively. This caused the trust score of the PWS to drop lower than the threshold value for this storm event. However, no observations were flagged by the PWS QC method in this event because none of the observations in this storm event met the predefined filter threshold of FZ, HI, and SO flags.
In a second example using PWS KTXHOUST443 (cluster 2) and storm event June 25, 2017 (Figure 8), the rainfall reported from this station was much higher than the neighboring consensus, which resulted in a decrease in the trust score because of some time steps of negative outcomes identified by the RSCRN. However, in this particular storm event, the actual consensus of rainfall observations might be uncertain because greater spatial variability existed across PWSs in the cluster. Based on the RSCRN algorithm, this was interpreted to mean the PWS was untrustworthy because of overreporting, which deviated from the consensus. However, as larger rainfall variability exists on this time step, the actual trustworthiness of this station might be uncertain. Further work should explore the role of rainfall variability within a cluster, and not just the robust average, in assigning negative outcomes in RSCRN.

Validation Using High-Fidelity Rainfall Stations
To validate whether RSCRN can result in higher accuracy of PWS-derived rainfall estimates, the RMSE between rainfall observations from PWSs to high-fidelity rainfall stations (HCFCD) at storm events was computed. As shown in Figure 2, the HCFCD rain gauges (445 and 435 in cluster 1, 520 and 560 in cluster 2) were in close proximity (mostly less than 1 km) with PWSs in the clusters and thus were used as the ground truth of actual rainfall observations for validation. The forgetting factor λ was set to 0.95, and the trust score threshold γ was set to 5.0 in this validation, determined by the best performance of the resulting RMSE.
Tables 5 and 6 show the RMSE comparison of PWS rainfall estimates for the analyzed storm events. In these comparisons, the RMSEs were computed using all PWSs (Equation 14, denoted as R all ), trustworthy PWSs (Equation 15, denoted as R RSCRN ), and QC PWSs (Equation 17, denoted as R QC ). The resulting R all ranged from 0.43 to 3.41 mm across the analyzed storm events, except for the storm event March 5, 2017, in cluster 1 for which a single PWS (KTXHOUST355) reported an extreme value of 1462.53 mm, which resulted in much higher R RSCRN for this particular storm event (column 1 in Table 5). It is worth noting that, because the RSCRN method did not rely on only a single observation to determine the trustworthiness of a PWS, it did not identify this station as untrustworthy for this storm event, which resulted in worse performance at this particular storm event. This suggests, however, that RSCRN could be used along with basic outlier detection methods to improve its results.
The overall performances for both methods are shown in Table 7. Using the RSCRN method, the results showed that of the 33 analyzed storm events in cluster 1, R RSCRN outperformed R all for 25 (76%) of the events, with a median RMSE improvement (ΔRMSE) of 0.35 (24.5%) (bold values in column 4 of Table 5, which is computed by subtracting R RSCRN with R all ). Of the 29 analyzed storm events in cluster 2, R RSCRN outperformed R all for 23 (79%) of the events, with a median RMSE improvement of 0.41 (29.8%) (bold values in column 4 of Table 6). Using the PWS QC method, results showed that R QC improved for 21 (64%) of the events in cluster 1, and 17 (59%) of  Tables 5 and 6). This demonstrates that both approaches made significant improvements in PWS rainfall estimates for the majority of the storm events.
Comparison of R RSCRN and R QC showed that RSCRN generally outperformed the PWS QC method (Table 7). In cluster 1, the results showed that 2 (6%) storm events have the same performance. In the remaining 31 storm events, RSCRN outperformed the PWS QC method in 22 (71%) of the storm events, with a median RMSE improvement of 0.16 (11.5%) (shown in bold values in column 6 of Table 5), while the PWS QC method outperformed RSCRN in 9 (29%) of the storm events (shown in italic values in the column 6 of Table 5). In cluster 2, the results showed that 1 (3%) storm event had the same performance. In the remaining 28 storm events, RSCRN outperformed the PWS QC method in 22 (79%) storm events, with a median RMSE improvement of 0.26 (21.7%), while the PWS QC method outperformed RSCRN in 6 (21%) storm events. This suggests that the RSCRN approach identified additional untrustworthy PWSs that were not flagged by the PWS QC method, and thus improved the rainfall estimates from PWS network.

The Efficacy of RSCRN
The proposed RSCRN provides a framework for evaluating the trustworthiness of crowdsourced PWSs. The trust scores derived from RSCRN were demonstrated to reflect both the historical and real-time level of trustworthiness of a PWS for capturing rainfall during a storm event, which could be used to improve rainfall estimates from crowdsourced PWSs. However, the efficacy of the RSCRN can be affected by several factors. For example, RSCRN would generally perform better in populated regions with a larger number of PWSs in close proximity, while the algorithm would be less effective in regions with larger PWS inter-distances and fewer PWSs. In addition, the parameter choices can also affect the obtained results from the RSCRN algorithm. As discussed in Sections 4.1.1 and 4.1.2, using a higher forgetting factor λ ensures longer history of the observations were accounted for, but the resulting trust scores would be less sensitive to sudden changes. A higher trust score threshold γ would result in more PWSs being classified as untrustworthy, but one might risk excluding trustworthy PWSs that are actually contributing trustworthy rainfall observations. On the other hand, choosing a lower γ might result in failing to identify a PWS that is actually untrustworthy. Furthermore, the choice of these parameters will also  affect the lead-up time of the RSCRN to converge to a reliable trust score, which is an important factor for enabling the RSCRN to operate in real time. Therefore, these factors should be carefully considered and tested for a local region to ensure the efficacy of RSCRN for evaluating the trustworthiness of PWSs in a network in those regions. Finally, while this study focused on real-world data, additional studies using synthetic data would also be valuable to further test the limitations of RSCRN through controlled experiments with synthetic datasets.

The Assumption of Consensus in Rainfall Observations
One of the premises behind RSCRN is that consensus in crowdsourced rainfall observations exists at some scale in space and time and can, therefore, be used to judge trustworthiness of stations within a cluster. Such consensus-based ideas are widely used across disciplines to identify errors in data (Foody et al., 2018;Strobl et al., 2019;Zhang et al., 2017). Strong consensus in rainfall observations occurs when rain gauges are located in close proximity, but the exact distance and other factors that should be used for defining a cluster are uncertain. This idea is not new, however. For example, the United States Climate Reference Network (USCRN), an extremely high-fidelity rainfall network, uses three distinct tipping bucket sensors installed next to each other on the same site for immediate detection of single sensor failure (Diamond et al., 2013). Better accounting for factors that influence consensus in rainfall observations (e.g., geography, climate, observation frequency) are possible extensions to the approach used in this study. In addition, in this work, a cluster was identified based on a group of PWSs that were close to each other. However, large rainfall variability may exist even over short distances, especially for high frequency rainfall observations or if stations have large elevation differences. For example, this particular case study focused on Houston, Texas, which has only slight topographical variation across the region; thus elevation was not a factor in station clustering. However, for regions with greater variation in elevation such as mountainous areas, the clustering results should include elevation to reflect where the consensus actually exists (Buytaert et al., 2006). Rainfall types can also be one of the factors that affect the consensus. For example, a convective storm may produce rainfall over a small area that is only captured by a single PWS in a cluster if clusters are not carefully created. In these cases, incorporating additional variables of the PWS location into the clustering method may better capture the consensus and thus result in more meaningful trust scores.

Feedback to Data Collectors for Improved Crowdsourced Data Quality
People are motivated by various kinds of incentives to adopt a PWS. Examples of these incentives include obtaining useful weather data for personal purposes or having a sense of belonging to a community of friends with shared interests (Gharesifard & Wehn, 2016). As the need for higher spatial and temporal resolution of rainfall data increases, the role of PWS data may be shifted from serving personal interests to benefiting society at large (Gharesifard & Wehn, 2016). In this case, because people who need to use the data are interested in knowing the quality of the data they contributed, PWS owners might become what Jøsang et al. (2007) described as service providers. To manage their provision trust, they may be willing to demonstrate their competence in collecting data and arguably welcome any feedback to improve their data quality. As a result, RSCRN could assist by making the trust score information available to the PWS owners. PWS owners could be notified of a drop in the trust score, and actions could be taken to correct the erroneous observations (e.g., cleaning the clogged rain gauge). Such efforts not only help  restore the trust scores, which maintain their provision trust, but also greatly improve the overall data quality of the crowdsourced rainfall network in the long term. Future improvements to RSCRN could focus on identifying particular types of errors to more effectively advise users on steps to improve their trust score.

Limitation of Binary Trust Score Threshold
Trust scores derived from the RSCRN represent the relative frequency of a PWS reporting trustworthy rainfall observations in the future. This continuous form is computationally efficient for reputation systems to calculate and update over time (Ruan & Durresi, 2016). However, to better enable reputation-based decision making, a discrete format of trust scores is often used (Mousa et al., 2015), as humans are generally better able to understand discrete verbal statements than continuous measures (Jøsang et al., 2007). In this study, we used a RSCRN-derived trust score threshold approach to classify PWSs as either trustworthy or untrustworthy. This can be thought of converting a continuous measure of trust metric into a binary discrete format. While using a binary trust score threshold is simple and straightforward for enabling decision making (e.g., include or ignore rainfall from a specific PWS), it does not represent the varying trustworthiness of PWSs (Ruan & Durresi, 2016). For example, a PWS with an extremely low trust score and a PWS with a trust score just below the threshold will both be categorized as an untrustworthy PWS, despite the difference in the extent of their untrustworthiness. Alternatives can be dynamically adjusting the binary trust score threshold to optimize decision-making or use of multinomial discrete values such as very trustworthy, trustworthy, untrustworthy, very untrustworthy to account for a broader extent of trustworthiness across PWSs (Ruan & Durresi, 2016). Future work could explore extensions like this so that RSCRN is able to weight information from PWSs based on their trust score rather than simply including or excluding measurements using a threshold method.

The Potential of Crowdsourced Rainfall Data
While this study focuses on crowdsourced rainfall data collected from PWSs, the proposed RSCRN can be beneficial to ensure the trustworthiness of other emerging crowdsourcing rainfall data collection methods as well. Beyond the case of PWSs, recent improvement in crowdsourcing methods has further enabled rainfall observations to be collected from connected vehicles (Bartos et al., 2019), surveillance cameras (Jiang et al., 2019), and mobile phones (Guo et al., 2018). The availability of these crowdsourcing methods greatly facilitates more crowd-participation, but also raises concerns of increased uncertainty associated with data contributors, highlighting the need for evaluating the trustworthiness of crowdsourced data (Gharesifard & Wehn, 2016;Hunter et al., 2013).
In particular for fixed point observations such as PWSs and surveillance cameras, RSCRN can serve as a starting point for creating algorithms capable of systematically assigning the trustworthiness of these data based on physical principles able to be applied at scale for quickly growing networks.

The Availability and Reliability of Crowdsourced Data
PWS adoption has been growing rapidly thanks to the advancement of technologies that have made PWSs easy to install and affordable, as well as development of software able to connect and share the data through online  platforms. This increase in PWS data openly shared on the Internet has transformed the value of PWSs from serving the owners' interests to appealing to anyone in the broader community who might benefit from the information (Gharesifard & Wehn, 2016). However, PWS data accessibility depends heavily on the platform the PWSs are connected to. For example, the Weather Underground recently ended a freely available service of the Weather Underground API and replaced it with a new API service that allows only PWS contributors to utilize the service (WXForum, 2018a). The Weather Underground has also stopped the the automatic connection of PWSs of certain brands (e.g., Netatmo) to their platform (WXForum, 2018b), resulting in abrupt changes to the number of sensors available in the system. These kinds of sudden changes might happen in any crowdsourced platform without warning, which could further compromise accessibility and reduce the utility of crowdsourced data. The community would benefit from more standardization of open networks and data sharing agreements to make the most of this emerging data resource.

Conclusion
In this study, we presented a RSCRN for ensuring the trustworthiness of PWSs in a crowdsourced rainfall network. The RSCRN assigned trust scores to PWSs are calculated by (a) clustering the PWSs into groups with similar rainfall characteristics, (b) computing the rainfall observation consensus within each cluster using a robust average method, and (c) deriving trust scores using a beta reputation system.
Using PWS rainfall data collected from Houston, Texas, as a case study, we demonstrated how RSCRN is able to identify PWSs with untrustworthy rainfall data. By ignoring rainfall from untrustworthy PWSs using a RSCRN-derived trust score threshold, the accuracy of the resulting 15-min rainfall estimates better matched rainfall observations observed from high-fidelity rainfall stations for 77% (48 out of 62) of the analyzed storm events, with a median RMSE improvement of 27.3%. Compared to an existing PWS quality control method, results showed that while 3 (5.0%) storm events had the same performance, RSCRN improved rainfall estimates for 74% of the remaining storm events 44 out of 59, with a median RMSE improvement of 18.7%.
We return now to the research questions from Section 1 that guided this work to provide answers based on the research outcome.
1. How can we evaluate the trustworthiness of crowdsourced PWSs?
This study demonstrated that a reputation system approach could be useful in evaluating the trustworthiness of crowdsourced PWSs. Unlike a traditional QA/QC method, the reputation system approach collectively evaluates the trustworthiness of a PWS itself over time rather than from single observations collected at a gauge. The RSCRN presented in this study assigns trust scores to PWSs based on their agreement or disagreement with current and historical rainfall observations from neighboring PWS, and is able to converge to a confident trust score in 20-30 time steps, as well as accommodate sudden changes in trust levels.
2. To what extent could a reputation system approach improve rainfall estimates from PWSs?
The reputation system can be used to improve rainfall estimates in direct and indirect ways. First, the reputation system approach ensures the rainfall estimates were produced from trustworthy PWSs. Using threshold for RSCRN-derived trust scores, PWSs were classified as trustworthy or untrustworthy. By including only trustworthy PWSs in the rainfall estimation process, the resulting trustworthy rainfall estimates were greatly improved in accuracy when compared to rainfall observed from high-fidelity rainfall stations. Second, the reputation system approach has the potential to encourage PWS owners to maintain and contribute high quality data, which indirectly improves rainfall estimates from PWSs in the long term.
Future work could be aimed at (a) a larger analysis of crowdsourced rainfall networks to identify and quantify the extent of untrustworthy PWSs across cities and regions in the world, (b) enhancing the reputation system algorithm to account for rainfall variability in complex topography and finer-temporal scales, and (c) leveraging crowdsourced rainfall estimates to improve hydrological modeling such as rainfall-runoff and flood prediction. With a reputation system able to ensure the trustworthiness of PWSs and improve the quality of data collected through crowdsourced rainfall networks, this growing data resource can be more confidently adopted and trusted for water resources management and decision-making.