{'fraction_non_alphanumeric': 0.02810554198630844, 'fraction_numerical': 0.015377205958237298, 'mean_word_length': 5.113564668769716, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 1, 'https://': 5, 'lorem ipsum': 0, 'www.': 4, 'xml': 0}, 'pii_count': 3, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Useful Biodiversity Data Were Obtained by Novice Observers Using iNaturalist During College Orientation Retreats

Citizen scientist platforms such as eBird and iNaturalist are dramatically increasing the biodiversity data available for scientific research. Questions remain about the validity of data collected by people with undefined credentials. However, few studies have examined the data quality of citizen science studies in detail. As part of an autumn orientation program, the Honors College at UMass Boston invited incoming students for a retreat on Thompson Island in Boston Harbor Islands National and State Park. One of their activities was a three-hour bioblitz using iNaturalist. We reviewed data collected from three autumn orientations (2017–2019) to evaluate the quality of the data and to examine the hypothesis that first-time users can contribute useful biodiversity observations. The students collected more than 2,000 observations and uploaded more than 5,700 photographs, mostly of plants (about 50%) and animals (40%). Approximately 50% of the observations became Research Grade by iNaturalist criteria. Errors in GPS data (ca 1–4%) did not always place observations automatically in the project. First-time users, presumably because they are digital natives and have experience with cell phone cameras, quickly master the basics of iNaturalist. We conclude that students using the iNaturalist platform, with a crowd-sourced ID process, produce data that are useful for a variety of biodiversity studies. CORRESPONDING AUTHOR: Robert Stevenson University of Massachusetts Boston, US robert.stevenson@umb.edu


INTRODUCTION
The promise of citizen science (CS) holds that it can both educate its participants and provide useful scientific data (Bonney et al. 2009;Raddick et al. 2009;Bonney et al. 2016). Educators intuitively know this is a lofty goal for formal education. For centuries, societies have supported educational institutions where students must undertake many years of schooling to prepare themselves for careers in all kinds of disciplines, including science. Beyond the historical perspective, our own experiences as teachers affirms this challenge. Most often, science educators instruct students using well-thought-out exercises instead of doing authentic science (but see studies of class-based or course-based undergraduate research experiences such as Spell et al. 2017 andFlaherty et al. 2017). These carefully planned demonstrations and laboratory experiments are tested and revised so they "work." In contrast, the process of doing science is not straight forward, involving missteps and failed experiments (Understanding Science 2021) that necessarily slow down learning science content and that can be challenging to undertake in classrooms (Spell et al. 2017, Harlin et al. 2018. The fundamental dichotomy between learning and doing science makes it unsurprising that people question the validity of the data coming from CS programs (Cohn 2008;Dickinson, Zuckerberg, and Bonter 2010;Gura 2013). Indeed, CS programs are often challenged about the quality of the data in their projects (Aceves-Bueno et al. 2017).
There is no single approach CS programs use to engage citizens/students and ensure the quality of their data (Wiggins et al. 2011;Kosmala et al. 2016;Freitag, Meyer, and Whiteman 2016;Stevenson et al. 2021). CS projects that record biodiversity have a variety of approaches to ensuring data quality. The largest CS biodiversity program, eBird, is available to any user who signs up. It uses a multipronged system for data quality assurance. Elements of the eBird system include minimizing errors with data inputs, checking inputs in real time to prompt users about unusual observations, review by regional experts, and calibration of the sensing ability of individual observers Kelling et al. 2012Kelling et al. , 2015. The Coastal Observation and Seabird Survey Team (COASST) is geared toward local community members (Parrish et al. 2019). It has a strict selection process to recruit participants and a rigorous training program with extensive protocols to monitor dead seabirds and marine debris (Parrish et al. 2017). Seabirds carcasses are collected and shipped to experts for ID confirmation. Reef Environmental Education Foundation (REEF) engages the diving community to monitor marine habitats, especially those associated with fish in tropical waters around the globe. Recreational divers survey habitats using a rigorous monitoring protocol. Observations are reviewed by expert marine scientists. The program rates divers based on experience. Participants that achieve "expert" status can be invited to join in special regional monitoring and assessment diving expeditions (Schmitt, Wells, and Sullivan-Sealey 1998;C. Semmes, personal communication).
Overall, it is fair to say that strategies are tailored to the focus of the program and to the difficulty of the participants' tasks. Program design may include open enrollment or careful selection of participants, minimum or extensive time commitments, and little or extensive training, etc. The most evolved process to ensure data quality to date in the United States (USA) has been developed over decades by the water quality monitoring community. These CS projects require extensive planning documents called Quality Assurance Project Plans (QAPP) that are reviewed by state governments and the Environmental Protection Agency (EPA). Parrish et al. (2018) have provided a hierarchy of design criteria for collecting high-quality data in CS projects.
The elemental components of data needed for biodiversity studies are species observed, place of observation, and the date and time of observation. Recording place and time in a standard fashion can be difficult (Chapman 2005) but the adoption of smartphone apps for recording has reduced those errors. Identification of species is more difficult and can take specialists years to master and may require specialized equipment, if not DNA sequences, depending on the taxon. Image identification using neural networks algorithms has become practical in the past few years and is helping the ID process in apps such as Merlin, iNaturalist, and Pl@nt.Net. The success of eBird and REEF depend on the community of naturalists who have become experts at identifying birds and fish in the wild. Other CS biodiversity projects, such as iNaturalist, iSpot, and Pl@nt.Net assist with identification using a two step process. First, an observer takes a picture or makes a sound recording of the species in the field, so there is a digital record of the observation. The observations are then shared on the web, allowing species identification to be crowdsourced by experts around the globe in the second step. This CS design intentionally separates the collection of the data and the identification of the species so people collecting the data need not necessarily be able to identify a species. iNaturalist has been adopted by state agencies, the National Park Service, and the National Geographic Society for bioblitzes, and more recently, it has served as the backbone for the City Nature Challenge with bioblitzes now organized in over 200 cities around the globe. This paper is motivated by the questions of data quality generally and specifically data quality from first-time and novice users engaged in exploring nature as part of an educational retreat. The motive and context for this retreat are documented in a companion paper (Rokop et al. submitted). We undertook analysis of the data collected by students using iNaturalist to answer the question to what degree might students entering college with little or no experience observing nature make useful contributions to scientific efforts to characterize biodiversity in our own backyards and across the globe? We broke down this focal question into three components as follows: 1) Did the naive students using smartphones achieve photographs cataloged in iNaturalist of sufficient quality to allow species identifications? 2) Were the location data accurate? 3) Are the occurrence records (combination of species identification, observation location, and observation time) of sufficient quality that the records can be useful to scientists and park managers in the future?
There was no specific biodiversity question the students were asked to investigate beyond documenting biodiversity. On the basis of our analysis of the data, we have included a series of recommendations for program design to increase the value for biodiversity science.

METHODS
Each autumn from 2017 to 2019, two or three groups of 80 to 100 entering Honors students at the University of Massachusetts Boston were invited to attend a 2-day retreat on Thompson Island (https://thompsonisland.org/), part of Boston Harbor Islands National and State Park (https://www.nps.gov/boha/index.htm, and for history about the island, see bostonharborislands.org) One activity during this retreat was a three-hour bioblitz using the iNaturalist platform. The educational goal of this exercise was to allow the students to observe nature and to participate in a CS project (Rokop et al. 2021, submitted). Thompson Island is located about 1 km from UMass Boston and about 6 to 7 km from the center of Boston. It is managed by the Thompson Island Outward Bound Education Center, a nonprofit educational organization. The island is a drumlin, 69 hectares in area (170 acres) at high tide, roughly 2 km long and 400 meters across at its widest point, and 24 meters above sea level at its highest elevation. Thompson Island contains a diversity of habitats including open meadows, forests, salt marshes, and beaches.
Students were asked to download the iNaturalist app to their cell phones before coming to the island. On island, they were provided a brief (~30 minute) introduction to the project and use of the app by National Park Service Rangers. Subsequently, teams (including guides) spent 1.5 to 2 hours making observations. Roughly equal numbers of students were sent to different quadrants of the island and instructed to make as many observations of organisms as they could record with iNaturalist. An exception to this general set of instructions occurred in the first year when four teams were sent to each quadrant and asked to concentrate on searching for members of one of four groups of organisms (plants, insects, marine organisms, or fungi). After the first-year experience, instructors thought this was too restrictive for the students and abandoned the requirement in subsequent years.

EVALUATION OF OBSERVATIONS
iNaturalist evaluates observations with a three-category system of Casual, Needs ID, and Research Grade (See here for the official rating details). In addition to the iNaturalist evaluations, we tallied the number of photographs per observation and developed a rubric to score the quality of images as good, OK, or poor (See below for further explanation). We identified whether or not the observer tried to identify the species being observed, and we also evaluated the likelihood that an expert in a specific taxonomic group could ID the species using the images taken (and any recorded notes). We totaled the number of observations that were identified to the species and genus level by August 1st, 2020. Lastly, we evaluated the accuracy of GPS data, the assigned place names of the observations, and the time stamps of the observations. Image quality of an observation was determined for the group of photographs representing a specific specimen taken in aggregate (a recorded observation) using criteria of Table 1. It was important that a single species was clearly identifiable, properly lighted, and focused. If one or more photos included adjacent organisms, then at least one photograph must focus specifically on the selected species, and if there is only one photograph focusing specifically on the selected species, it must focus on aspects generally useful for taxonomic identification (examples are found in Figure 1 and Supplemental File 1: Supplemental Figure 1). Identification from photographs can be challenging but can be made more reliable by appropriate views of in-focus photographs. Capturing an image of the entire organism from a distance provides context for more detailed photographs. Observations were rated good for animals if multiple views were captured (lateral, dorsal, ventral), including images of appendages as appropriate. Likewise, a photograph or a group of photographs including an entire plant (above-  Table 1 Summary of the criteria to judge the quality of the collection of images for an observation and possibility of identification by an expert. The text contains additional information.

Figure 1
An example of a good (https://www.inaturalist.org/observations/31739760) (row 1, an oyster), OK (https://www.inaturalist.org/ observations/31742210) (row 2, a snake), and poor (https://www.inaturalist.org/observations/31740442) (row 3, a spider) set of images from three different observations. Criteria are based on Table 1. Reasons to score the third-row observation as poor are three more or less identical photographs with nothing specific for scale, with poor focus, and slightly obscured by the plastic container. A similar example is given for plants in Supplemental Figure 1.
ground aspects), branching pattern, leafing pattern, bark, and flowers or fruits (if present), was rated as good. For some species, this could be achieved in a single photograph and thus the image quality might be rated good if the observation included a single, in-focus photograph in which the intended species existed alone or was irrefutably the intended species (Figure 1). Such irrefutable intention was at times, although not often, made possible by explanatory notes recorded with the observation. In some instances, a single photograph or a collection of photographs was sufficient for identification of a commonly recognized species, but the observation was still rated poor or OK because it alone, or the aggregate photographs included, did not satisfy the above criteria. Thus, quality rating of an observation and its usefulness for identification were not necessarily tightly coupled. An OK rating indicated that the observation partially satisfied the criteria, but could easily be improved by better or additional photographs. Inclusion in an image of an object of known size (approximate or precise) generally improved ratings of an observation. During the course of the rubric development and scoring, we discovered that some observations made by students during the orientation program were not included in the project by the iNaturalist algorithm, a possibility we had not originally considered. As a consequence, we gathered and analyzed these additional observations located on the basis of user name, time frame, and GPS location. After completing these extra steps, we were confident that we had included all of the observations made by all users who were part of the orientation program. For analyses we relied on iNaturalist tools, Excel, Google docs, R (R Core Team, 2020), RStudio (RStudio Team 2019), and the rinat and iNatTools libraries.

RESULTS
Over three years (2017-2019) of the orientation program, a total of 468 students collected more than 5,600 images that were distributed across more than 2,000 observations, > 600 of which achieved Research Grade on iNaturalist (Tables 2 and 3). One new local species record, for Dekay's Brownsnake, was obtained.

USE OF THE INATURALIST PLATFORM
Roughly 700 observations were collected each year using approximately 85 first-time user accounts ( Table 2) that documented about 100 species ( Table 3). The numbers given in Table 2 include Park Service employees as well as students. Readers should be aware that we found that all the numbers in Table 2 changed over time because of data curation by project managers, individuals withdrawing their observations, and more identifications being made over time.
There were almost twice as many student participants the first year compared with the 2nd and 3rd years, but the number of user accounts varied little ( Table 3). The number of student observations and species identified increased slightly in each subsequent year ( Table 3). The groups in the second and third years were comparable in their productivity, with each making about 9 observations per account and identifying 1.5 species (Table 3).

OVERVIEW OF BIODIVERSITY DOCUMENTED
In the first two years, most observations were of plants (about 50%) and animals (40%), with the remaining 10% divided between fungi and algae (Figure 2). For 2019, the number of observations of fungi and algae remained the same as in prior years, the number of plant and animal observations increased, and the not available (NA)   category dropped substantially relative to the first two years (Figure 2). NA might indicate that identity of the observation was difficult or not possible, images were of humans or of objects not organisms, several organisms were included in the image with insufficient information to determine intention, etc. A total of 202 unique species were identified across the three years from the student orientation programs with 19% of the species common to all 3 years (Figure 3). Taxonomic diversity was high and included more than 90 families and about 100 species each year (based on analysis of iNaturalist data). Approximately 34% of the observations were identified to species level and thus achieved Research Grade. There was much more success with identification of animals and plants than of fungi and chromista (algae).
Among animals, the most commonly identified species were marine mollusks, crustaceans, and insects (Figure 4). These included hermit crabs, green shore crabs, blue mussel, common periwinkles, acorn barnacles, and European flat oysters in the marine habitats; and butterflies, bumblebees, grasshoppers, and spiders in the terrestrial habitats. As with animals, the students recorded or observed the most obvious plants, such as colorful herbaceous flowers or shrubs that were flowering or fruiting in the late summer.

NOTABLE BIODIVERSITY OBSERVATIONS
Of the more than 200 species observed by students on the island, four struck us as immediately noteworthy. First, a student caught and photographed the Dekay's Brownsnake (Storeria dekay). A search of iNaturalist found this to be the first and only observation of this species on the island (R. Vincent, personal communication). Second,   Year-to-year variation in the types of organisms the students documented is large. Across the years, crabs, mollusks, and butterflies were common among animals and shrubs, and wildflowers were common among plants.
there is a thriving population of English Oak (Quercus robur) that have naturalized the island. Quercus robur occurs across the USA and Canada and throughout New England (GoBotany 2021, iNaturalist 2021), but the largest and densest number of individuals in Massachusetts appears to be on Thompson Island (iNaturalist 2021). Lastly, students documented a large number of two invasive crab species, the European green crab (Carcinus maenas) and the Asian shore crab (Hemigrapsus sanguineus). The green crab has been on our Atlantic coast for 200 hundred years, whereas the Asian shore crab arrived in New Jersey in the 1980s and is now extending its range and population density along our coast (Bailie and Grabowski 2018). These data are likely to contribute to our understanding of how these species are interacting over time and space.

DATA QUALITY
Students were encouraged to photograph more than one view/character of the species they were observing, and indeed they did so. Sixty percent of the observations have 2 or 3 images, while the amount with just one image was between 10% and 20%.
Aspects of data quality useful for species identification improved over the three year period and especially between the first and the second year. Observers increased the number of observations they tried to identify from 2017 to 2019 (Figure 5a). The "Research" and "Needs ID" observation grades increased, and the NA grade decreased during the course of the study (Figure 5b), with about ⅓ of the observations achieving Research Grade.
We further asked, "Is it possible to identify this observation to species with the images provided?" In 55% to 70% of the cases we thought it was possible, meaning we think that with more attention by expert identifiers on the ID process, more observations can attain Research Grade (Table 1; Figure 6a). We found that the quality of the collection of images for each observation ( Table 1) was mainly OK or good in 85% of the observations; 15 % were poor (Figure 6b).
Location data were accurate as judged by two criteria. When plotted, observations follow expectations for terrestrial species (Figure 7, top panel) found across the island, mostly along walking paths, while marine species were observed along the shoreline (Figure 7, bottom panel). A few of the marine observations appear in deeper water indicating larger location errors. Those at the lowest latitudes (< 42.31°N) may represent observations made at low tide. The mode and median positional accuracy were good at 5 and 6 meters respectively (Figure 8).
Three issues compromised the quality of the location data. First, as noted above, when reviewing project data  by user ID, we discovered that some observation records were not recorded as being on Thompson Island when they should have been, owing to low GPS accuracy. These observations were recovered by searching for observation records within the appropriate time window from participating contributors. In 2017, 4.7% (n = 29) of the observations collected were not included. The corresponding values for 2018 and 2019 were 4.3% (n = 27) and 0.6% (n = 5). Second, occasionally there was large uncertainty about an observation, leading to a long tail to the distribution and a mean of 41 m (Figure 8). Third, there were more than 60 different place names given in the data set, making us initially question the location of an observation even though the vast major of GPS coordinates accurately located observations on Thompson Island.
We found no discernable errors in the date-time data. It was our confidence in the date-time date that allowed us to discover the observations that did not make it into the projects described in the previous paragraph.

IDENTIFICATION PROCESS
The identification process of iNaturalist is key to its success. The process allows anyone willing to review images the opportunity to help identify organisms. In all, there were 6,665 identifications made by 713 different identifiers over three years. The number of unique reviewers by year was 269, 319, and 312. The identification process revealed that most identifiers helped refine identifications.

BIODIVERSITY FINDINGS
Our naive users collected more than 5,600 images, which resulted in the identification of 202 species and potentially more. A year-long biodiversity inventory in 2017 of the Boston Harbor Islands using the iNaturalist platform found 475 species. By comparison, the 202 species identified on Thompson Island represent the short, late-summer sampling period and the three-hour-per-year limit to the activity. This short field experience contributes to the relatively low (19%) proportion of species common to the  three years (Figures 3 and 4). The students were predictably attracted to species that were easily photographed (e.g., did not move or were of the right size). Examples include herbs and shrubs that were flowering or fruiting, oysters, mussels, snail shells, and insects such as butterflies. Nonetheless these observations build a permanent (assuming they are not withdrawn) and accessible record that can contribute to future studies of biodiversity such as species distributions, invasive species, phenology, and population studies.
Such data could also be of value to future researchers with more focused interests.

WHAT IS THE QUALITY OF THE DATA COLLECTED?
As noted in the Introduction, a common data-quality issue in biodiversity records is taxon identification. Because of the diversity of the information in the records, differences in the identification and review process, and because the taxa vary widely across projects, a logical measure of data quality would be to compare the overall percentage of Research-quality observations in the Orientation Retreat project with the percentage found across the entire iNaturalist platform. We expected the percentage for this project to be lower because the participants are mostly firsttime users and because we have more plant than animal observations, whereas in iNaturalist, animal observations are more common than plant observations. The percentage of Research Grade observations in iNaturalist is 40.8 % (based on 23.0 million observations that are of Research Grade in GBIF and 56.4 million observations overall on the iNaturalist website), whereas the identification process for the Orientation Retreat projects has converted just 34.0% (Figure 5) of the observations to Research Grade. It seems that the observations provided by these first time users were advanced to Research Grade at approximately 80% of those of the iNaturalist community as a whole.
Although there are many ways that images of a species can be improved, including image focus, documenting the whole organism and its parts, and capturing details essential for identification, most of these aspects depend on experience and knowledge of the species being observed.
One of the benefits of smartphone use is the recording of precise GPS location and the date and time of the observation. Date-time data were precise. We also expected that using cell phone location technology, at a study site within a few miles of a major metropolitan city such as Boston, would lead to precise and accurate species locations, and that was what we generally found (Figures 7 and 8). However, the location data depends on the cell phone signal provider and relative tower location.
Location errors associated with an observation can be large, and consequently, 2.7% of observations averaged over three years that should have been included in the projects were initially not included, and the location uncertainty for observations was greater than 10 m for 30% of all observations (Figure 8). We also found that the location names in the data files for this coastal/marine environment caused confusion. It is important for people analyzing location data to keep these potential uncertainties in mind.

RECOMMENDATIONS FOR PROGRAM DESIGN
We undertook analysis of the data collected by students using iNaturalist to answer the questions "Can and under what conditions might students entering college with little or no experience observing nature make useful contributions to characterize biodiversity in our own backyards?" We found that first-time users can make useful contributions. We believe a combination of factors work synergistically for success. These include: 1) the separation of the collection of data from the identification process on the iNaturalist platform, 2) the superbly designed interface of iNaturalist providing a minimum threshold for students to start to contribute, 3) the number of sensors (built in cameras and GPS) and computer power of the current generation of smartphones, 4) the years of experiences students have with using smartphones, 5) the species image-recognition software available in iNaturalist, 6) the initial training sessions and guidance during the session, and 7) project curation after collection within the iNaturalist database.
As in most endeavors, there are several aspects that can be improved by: 1) gaining better skills at taking close-up images, 2) paying more attention to scale and parts of different species, and 3) increasing the use of the improving capability of species-identification software in iNaturalist, which was clearly enhanced over the 3-year study.

CONCLUSION
The iNaturalist Honors College retreat exercise was designed to get incoming students outside to observe nature and to engage with a CS project (Rokop et al. submitted). In addition to this educational goal, we investigated the scientific contribution of the bioblitzes. College students, with their years of experience using smartphones, can, with only brief instruction, collect and post observations to the iNaturalist platform. We conclude that these students using iNaturalist, with its crowdsourced ID process, have produced data that are useful for a variety of biodiversity studies.

DATA ACCESSIBILITY STATEMENT
Data available through the iNaturalist project pages as listed in the paper and provided in Supplemental Files 2 and 3.

SUPPLEMENTARY FILES
The supplementary files for this article can be found as follows: • Supplemental File 1: Supplemental Figure 1. This files gives images of plants to illustrate the grading system for images (Table 1)