{'fraction_non_alphanumeric': 0.029876765855124738, 'fraction_numerical': 0.035347159603246164, 'mean_word_length': 5.1071953010279, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Thinking like a naturalist: enhancing computer vision of citizen science images by harnessing contextual data

The accurate identification of species in images submitted by citizen scientists is currently a bottleneck for many data uses. Machine learning tools offer the potential to provide rapid, objective and scalable species identification for the benefit of many aspects of ecological science. Currently, most approaches only make use of image pixel data for classification. However, an experienced naturalist would also use a wide variety of contextual information such as the location and date of recording. Here, we examine the automated identification of ladybird (Coccinellidae) records from the British Isles submitted to the UK Ladybird Survey, a volunteer-led mass participation recording scheme. Each image is associated with metadata; a date, location and recorder ID, which can be cross-referenced with other data sources to determine local weather at the time of recording, habitat types and the experience of the observer. We built multi-input neural network models that synthesise metadata and images to identify records to species level. We show that machine learning models can effectively harness contextual information to improve the interpretation of images. Against an image-only baseline of 48.2%, we observe a 9.1 percentage-point improvement in top-1 accuracy with a multi-input model compared to only a 3.6% increase when using an ensemble of image and metadata models. This suggests that contextual data is being used to interpret an image, beyond just providing a prior expectation. We show that our neural network models appear to be utilising similar pieces of evidence as human naturalists to make identifications. Metadata is a key tool for human naturalists. We show it can also be harnessed by computer vision systems. Contextualisation offers considerable extra information, particularly for challenging species, even within small and relatively homogeneous areas such as the British Isles. Although complex relationships between disparate sources of information can be profitably interpreted by simple neural network architectures, there is likely considerable room for further progress. Contextualising images has the potential to lead to a step change in the accuracy of automated identification tools, with considerable benefits for large scale verification of submitted records.

. Nonetheless, at present, general-purpose 48 automated classification of animal species is currently some distance from the level of accuracy 49 obtained by humans, and the potential remains underutilised. 50 The large data requirements and capacity of machine learning has led to a close association with 51 citizen science projects , where volunteers contribute scientific data 52 (Silvertown, 2009). Citizen scientists can accurately crowd-source identification of researcher-gathered 53 images (e.g. Snapshot Serengeti; Swanson et al., 2015), generate records to be validated by experts 54 (e.g. iRecord; Pocock, Roy, Preston, & Roy, 2015) or both simultaneously (e.g. iNaturalist; 55 iNaturalist.org). However, there can be a considerable lag between record submission and human 56 verification. If computer vision tools could generate more rapid, or even instantaneous, identifications 57 it could assist with citizen scientist recruitment and retention. While image acquisition by researchers 58 can be directly controlled and lead to high accuracies ( Most automatic species identification tools only make use of images (Weinstein 2018). However, an 62 experienced naturalist would utilise a wide variety of contextual information when making an 63 identification. This is particularly the case when distinguishing 'difficult' species, where background 64 information about the record may be essential for a confident identification. In a machine learning 65 context, this supplementary information about an image (metadata) can be split into two categories 66 ( Figure 1). Primary metadata is directly associated with a record such as GPS-coordinates, date of 67 recording and the identity of the recorder. Derived (secondary) metadata is generated through cross-68 referencing with other sources of information to place this metadata into a more informative context 69 (Tang, Paluri, Fei-Fei, Fergus, & Bourdev, 2015). In an ecological context, this may include weather 70 records, maps of species distribution, climate or habitat, phenology records, recorder experience, or 71 any other information source that could support an identification. directly associated with an image such as the date or location. By contrast, derived (or secondary) metadata 76 requires cross-reference to external databases, which may include physical, ecological or social data. External 77 sources of information may be fixed and stable (such as habitat maps) or dynamic and require updating in order 78 to keep the model up to date (such as weather records or recorder experience).

79
Efforts to include contextual spatio-temporal information have largely focused on reducing the list of 80 potential species that may be expected in a given area. iRecord (www.brc.ac.uk/irecord) partially 81 automates this process, flagging records to expert verifiers that are labelled as being outside of the 82 6 therefore represents an example of a programme where a reliable automated identification tool could 116 help to increase the use of citizen science to document biodiversity across the globe. 117 Classification tools that only use image data are not making maximal use of the information available 118 to human experts. Here we demonstrate methods to incorporate metadata directly within neural 119 networks used for the classification of images of ladybirds submitted to the UK Ladybird Survey. We 120 examine if metadata can significantly improve classification accuracy, thereby increasing their 121 potential to assist in large-scale biodiversity monitoring, by: 122 1. Comparing the classification accuracy of classifiers incorporating metadata compared to image-only 123 classifiers. 124 2. Exploring whether neural networks make use of the same pieces of metadata information that a 125 human experts do. 126

Methods 127
Data 128 Records of ladybirds (Coccinellidae) were sourced from the UK Biological Records Centre 129 (www.brc.ac.uk). These were filtered to include only those from within the British Isles, from 2013 to 130 2018 inclusive, that contained an image and had been verified by an expert assessor. Records were 131 distributed across the whole of the British Isles, although records were more frequent near more-132 heavily populated areas ( Figure S1). The date range was selected based on a notable increase in 133 records from 2013 with the release of a mobile app (iRecord Ladybirds). Identifications of records by 134 expert verifiers was based on uploaded images and associated information including the species 135 determination of the original observer, location, date, associated comments and (where known) the 136 degree of skill of the recorder. 137 Of the 47 species of ladybird that had been recorded at least once in the UK (Duff, 2018), only 18 138 species (listed in table 1) had at least 170 usable records, which we took as our lower cut-off to ensure 7 each species was represented by at least 120 unique training images. We judged that fewer training 140 images would not result in accurate classification. These 18 species made up 97% of the total ladybird 141 records during 2013-2018. Even after removing species with fewer than 170 usable records, the data 142 set is highly imbalanced (Table 1), with two species making up the bulk of records: 7-spot ladybird 143 Coccinella septempunctata (25.8%) and the highly polymorphic Harlequin ladybird (44.5%). 144 Images 145 Records were manually scanned to remove the majority of images predominantly of eggs, larvae or 146 pupae, 'contextual' images of habitat area, images including multiple species, and images that had 147 been uploaded repeatedly. Larval and pupal images were overwhelming dominated by the highly 148 distinctive Harlequin ladybird larvae or pupae (78%). Where a single record had multiple associated 149 images, only the first was used. Images were centre cropped to square and then rescaled to 299x299 We constructed models that made use of different subsets of the available metadata. The first (the 163 primary metadata model) took only three pieces of primary metadata, drawn directly from the UK 164 Ladybird Survey dataset: longitude, latitude and date. We represented date by day-of-year, excluding 165 year values since information on 'year' would not be transferable to future records. The second model 166 (the derived metadata model) supplemented the primary metadata with secondary metadata: data 167 generated with additional reference to external sources of information, namely weather records, 168 habitat and recorder expertise. We did not use the original citizen scientist species determination in 169 our models, since it was too powerful compared to other sources of information (correct over 92% of Local habitat information was derived from a 1km resolution land cover map (Rowland et al., 2017). 177 This provides percentages in each 1km grid of 21 target habitat classes (e.g. 'urban', 'coniferous 178 woodland', 'heather', etc.). Where no data was available, each habitat was assumed to be 0. 179 We calculated a 'recorder experience' variable as the cumulative count of records submitted by that 180 recorder at the time of each record. Only records of ladybirds in our dataset were included in this 181 count. Where no unique recorder ID was available, that record was assumed to be a first record. are highly skewed towards certain common species (Table 1). As predictive models are not perfect, 225 such class-imbalanced data leads to critical choices about how to best assess 'accuracy'. Overall 226 accuracy may be maximised by rarely or never assigning species to unusual categories. A citizen 227 scientist may prefer the maximum accuracy for the species in front of them (which is likely to be a 228 commonly reported species). However, in an ecological science context, rare (or more precisely, rarely 229 reported) species are often of particular interest to researchers managing citizen science projects. 230 The total dataset was randomly partitioned into training (70%), validation (15%) and test (15%) sets. 231 To address the class-imbalance, we followed the approach suggested by Buda, Maki, & Mazurowski, 232 (2018) and re-balanced our training set through up-sampling and down-sampling the available 233 records. We did this so that each species had 2000 effective training records. Consequently, our 234 underlying models did not have direct access to the information that, all else being equal, certain 235 species are far more likely than others. This reduces the potential for the model 'cheating' during 236 training by fixating on common species and ignoring rare species. To demonstrate the potential to 237 improve overall accuracy by taking into account the relative frequency of each species, we tested 238 weighted versions of each of the models. In these, the relative probability assigned to each species 239 from each unweighted model ( ) were scaled by the relative frequency of each of the species ( ) in 240 the training data as:

Model Testing and Evaluation 255
Overall and species-level model performance was assessed in terms of top-1 (was the true ID rated 256 most likely) and top-3 (was the true ID amongst the three options rated most highly) accuracy. 257 Because model accuracy will be dependent on the split of data into testing and training sets, and 258 because model optimisation is a non-deterministic process, we repeated the entire model fitting 259 process 5 times. For each repeat, assignment of images to training, validation and test sets was 260 randomised. 261

Role of Metadata Components 262
To examine the dependence of the model on each aspect of the metadata we examined the decline in 263 top-3 accuracy for each species when elements of metadata were randomised by reshuffling sets of 264 values within the test set. We did this separately for the spatial coordinates, day-of-year, 265 temperatures data, habitats data and recorder expertise. 266

267
Across each of our training-test split realisations, combined multi-input models showed a marked and 268 consistent improvement on both the image-only (+ 9.1 percentage points) and the ensemble models 269 (+ 3.6 percentage points) (Figure 4). Species-level accuracies (averaged across the 5 split realisations) 270 for each of the models are reported in Table 1

284
The overall accuracy of all models could be greatly improved by weighting the output probabilities by 285 the prior expectation given the relative frequency of each species. For example, the average top-1 286 accuracy of the combined model rises from 57% to 69%. However, these gains are made at the cost of Where both temperature and day-of-year data was available, the temperature data appears to be 302 used more (10% and 0.2% decreases respectively). It is not possible to determine whether this is 303 because temperature is simply more relevant to ladybirds than date, or whether this is an artefact of 304 the different lengths of the metadata vectors. When day-of-year was randomised in the primary 305 metadata model, top-3 accuracy declines by 4.5% points. Within temperature, the model appeared to 306 be making more use of the weekly temperature data (2-10 weeks before the record), where 307 randomisation caused an 8.1% decrease than the more proximate daily records for the preceding 308 fortnight (-5.4%). The remaining metadata components had smaller influences on overall top-3 309 accuracy: randomising habitat data led to a 2.8% decrease while randomising recorder experience led 310 to a 2.1% decrease. 311 These overall results are highly influenced by the dominant species (particularly the Harlequin 312 ladybird) in the test set, masking variation in decline in accuracy on a per-species level (SI Table S2). 313 The apparent importance of each metadata component appears to align with ecological expectations. 314 The five species with greatest decline in accuracy when habitat is randomised are all considered 315 habitat specialists (Roy & Brown, 2018): Coccinella undecimpunctata (dunes), Anatis ocellata 316 (conifers), Tytthaspis sedecimpunctata (grassland and dunes), Subcoccinella vigintiquattuorpunctata 317 (grassland), and Aphidecta obliterata (conifers). Similarly, the randomisation of location had the 318 greatest effect on the localised species ( Figure S1). The top three most affected were: Aphidecta 319 obliterata (frequently reported in Scotland), Scymnus interruptus (South-East England) and Coccinella 320 undecimpunctata (coastal). By contrast, the Seven-Spot ladybird, a widespread and generalist speices 321 was poorly identified by the metadata model and showed a minimal response to randomisation. The 322 species affected most by the randomisation of temperature was Propylea quattuordecimpunctata, 323 with the common name of the 'dormouse' ladybird (Roy and Brown 2018, p.112) because of its known 324 late emergence. 325 The randomisation of recorder experience had the greatest impact on Scymnus interruptus. This was 326 the only 'inconspicuous' ladybird in our dataset, which inexperienced recorders may not even realise is 327 a ladybird (see Figure 2g). There was also a 10% decrease in the identification of Harlequin ladybirds 328 when recorder experience was randomised. Novice recorders are notably more likely to record 329 Harlequin ladybirds than more experienced recorders. The first record submitted by a new recorder is 330 a Harlequin ladybird 57.4% of the time, which rapidly declines to 38% by the 10 th . 331

Discussion 332
The use of metadata within computer vision models considerably improves their reliability for species 333 identification. This exciting finding has implications for biological recording, demonstrating the 334 18 potential to use innovative approaches to assist in processing large occurrence datasets accrued 335 through mass participation citizen science. Basic primary metadata is straightforward to incorporate 336 within machine learning models and, since this information is already collected alongside the 337 biological records, can be widely adopted. 338

Interpretation of results 339
The notable gain in accuracy of the combined multi-input model compared to the ensemble model is