{'fraction_non_alphanumeric': 0.03074675221548137, 'fraction_numerical': 0.012991138074516406, 'mean_word_length': 4.905833802288501, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Investigating the Cognitive Processes Involved in Cancer Cell Image Identification

The ability to make timely and accurate decisions is critical for cancer detection in diagnostic pathology. We describe a joint experimental and computational modeling approach for determining the impact of three factors on the identification of peripheral blood blast cells (cancer indicators) versus normal white blood cells in microscopy images of clinical samples: (1) image difficulty, (2) time pressure, and (3) externally imposed bias. Our study also examines how performance on accurate identification of blast cell images is related to general object recognition ability (as measured using the Novel Object Memory Test (NOMT) (Richler et al., 2017). For this study we collected a bank of hundreds of digital images that were identified by cell type and classified by difficulty by a panel of expert hematopathologists. Participants in this study were a non-expert, student population. The Diffusion Decision Model was fit to choice and response time results of this experiment to determine how these factors influence decision processes. Results indicate that external time pressure leads to a reduction in caution in participants while external bias has a small but detectable influence on choices. Comparison of cancer image identification results with NOMT results indicates participants with stronger general object recognition ability perform better on the cell identification task, but only on easier images. This indicates that general object recognition ability carries over to this diagnostic task to a limited extent.


Introduction
Accurate interpretation and classification of medical images is an important step in the diagnosis and treatment of numerous diseases. A wide range of medical disciplines (Samei & Krupinski, 2010) ranging from pathology (our focus here), to radiology, to ophthalmology rely on expert analysis of images to detect abnormalities. While the exact rate of diagnostic errors is unknown, consistent evidence suggests error rates are >10% (Goldman et al., 1983;Hoff, 2013;Kirch & Schafii, 1996;Shojania, Burton, McDonald, & Goldman, 2003;Sonderegger-Iseli, Burger, Muntwyler, & Salomon, 2000). It is thus critical that we understand how people make perceptual decisions from medical images in order to improve training and minimize the occurrence of misdiagnoses. This requires investigation of the cognitive processes underlying decision-making in this domain and how those processes are manipulated by external factors. For example, how does time pressure influence behavior? Alternatively, how do externally imposed biases influence performance? After all, in many cases, images have passed through a computer aided diagnosis process and / or analysis by medical technologists and residents before being verified by a senior clinical expert.
Decisions based on medical images have a number of parallels with perceptual decision-making, where people make choices based on sensory information. The investigation of perceptual decision-making has a rich history in psychology, cognitive science, and neuroscience (Gold & Shadlen, 2007;Heekeren, Marrett, & Ungerleider, 2008;Summerfield & de Lange, 2014;Summerfield & Egner, 2009). However, perceptual decision-making of medical images in clinical settings has received less attention. This is in part due to several logistical challenges not typically present in lab settings (Wolfe, 2016). First, the collection of an annotated bank of images requires significant time investment from experts to classify those images. Second, time and access become issues if you are looking to experts to participate in experimental studies: there are usually few local experts in a particular diagnostic specialty (low sample size) and they are usually very busy. Third, manipulating cognitive variables or aspects of the experiment can be challenging. Thus it is important to devise strategies for investigating this domain in both clinical and non-clinical settings, and to find ways to connect the larger body of prior work on lab based perceptual decision-making to the clinical context.
Here, we present a study investigating the cognitive processes underlying cancer image detection in diagnostic pathology. More specifically, we investigate how various external factors influence the ability of non-expert undergraduate students to distinguish between normal (standard white blood cells such as lymphocytes, monocytes, or neutrophils) and abnormal cancer cells ("blast" cells, associated with acute leukemia) in clinical images. Toward this end, we take a joint experimental and modeling approach utilizing experimental paradigms and modeling methods previously developed in the course of basic research on perceptual decision-making (Ratcliff & Smith, 2004;Schouten & Bekker, 1967;Wickelgren, 1977).
To investigate this process experimentally, we passively collected a large bank of digital images of both blast and non-blast white blood cells drawn from patients at the Vanderbilt University Medical Center (all images were obtained as part of routine clinical care). A panel of expert pathologists classified each of these images, providing a fully annotated data set consisting of hundreds of images of varying type and level of difficulty. Using this image bank, we developed a speeded decision-making experiment to investigate how difficulty, time pressure, and externally imposed bias influence individuals' behavior.
In addition to testing participants' ability to discriminate between and classify images of blast and non-blast cells, we also investigate how participants' general object recognition ability impacts their performance on this task. Toward this end, we employ a second task, the Novel Object Memory Test (NOMT), to assess each participant's general ability to learn and recognize objects that they have no prior experience with (Richler, Wilmer, & Gauthier, 2017). We use this to probe to what extent general object recognition, which has been studied in much more detail in lab settings (Gauthier et al., 2014;Hildebrandt, Wilhelm, Herzmann, & Sommer, 2013;McGugin, Gatenby, Gore, & Gauthier, 2012;McGugin, Richler, Herzmann, Speegle, & Gauthier, 2012), correlate with or impact participants' efficacy on the blast cell identification task.
To gain further insight into the cognitive processes underlying decisions on this task, we utilize computational modeling linked with results of this experiment. One of the benefits of quantitative modeling, and the reason we use it here, is that it provides a way to quantify latent cognitive processes and statistically separate the different components of the decision process (caution, bias, and rate of information uptake) that are not accessible through traditional statistical methods alone. For this, we utilize a version of the classic Diffusion Decision Model (DDM) (Ratcliff, 1978;Ratcliff & McKoon, 2008;Ratcliff, Smith, Brown, & McKoon, 2016), which has been shown to account for detailed patterns of behavior across a wide range of speeded decision-making paradigms (Ratcliff, Love, Thompson, & Opfer, 2012;Ratcliff, Thapar, & McKoon, 2001, to model the choice and response time behavior of participants on this task and extract these underlying cognitive parameters.

Experimental Methods
Participants. 37 undergraduate students at Vanderbilt University participated in exchange for course credit. We set a target sample size of about 35 participants. This sample size was selected on the basis of previous experiments conducted by the authors that used similar modeling methods (Holmes, Trueblood, & Heathcote, 2016). A ratings panel of three hematopathology faculty from the Department of Pathology at Vanderbilt University Medical Center was used to identify and rate each image. The raters first identified each image as a blast or a non-blast cell. Following this identification, they were asked to provide a difficulty rating for each image on a 1-5 scale. If the raters identified the image as a blast cell, they were asked to rate how similar the image was to a classic blast cell (with a rating of 1 being 'not similar' and a rating of 5 being 'very similar'). Raters were told that a classic blast cell image is one that might be used in a textbook. If raters identified the image as a non-blast cell, they were asked to rate how morphologically similar the cell is to a blast cell (with a rating of 1 being 'not similar' and a rating of 5 being 'very similar').
Out of the original set of 840 images, the three expert raters agreed on the cell type (i.e., blast or non-blast) for 633 images. From this set, we grouped the images into four types based on the difficulty ratings. The average rating for the blast images was 4.40 (SD= 0.46) and the average rating for the non-blast images was 1.68 (SD = 0.55).
Blast images that received an average rating of 4.5 or greater were categorized as easy blasts (151 images, Fig 1a). Blasts images that received an average rating of 4 or less were categorized as hard blasts (98 images, Fig 1b). Images with an average rating between 4 and 4.5 were not included. Non-blast images that received an average rating of 1.5 or less were categorized as easy non-blasts (129 images, Fig 1c). Non-blasts images that received an average rating of 2 or greater were categorized as hard non-blasts (108 images, Fig 1d). Images with an average rating between 1.5 and 2 were not included. For the experiment, we selected 75 images from each category for a total of 300 unique images. Note that we have more than 300 trials in the experiment, so some images are repeated. However, no images were repeated until all of the images from a category had been shown. Thus, we doubt participants were able to recognize that images were shown more than once.
Procedure. In the main task, participants first completed a training stage to familiarize themselves with blast cells. There were four blocks of training trials. Each block started with participants studying five images of blast cells one at a time. After studying these five images, participants then completed 15 trials where they were presented three images (1 blast image and 2 non-blast images) and asked to choose the image they thought was the blast cell. The four training blocks had the following structure of blast and non-blast images: block 1 was easy blast versus easy non-blast, block 2 was easy blast versus hard non-blast, block 3 was hard blast versus easy nonblast, and block 4 was hard blast versus hard non-blast.
After completing the four training blocks, participants completed a practice block of 60 trials to familiarize themselves with the main task. Each trial started with a fixation cross displayed for 250 ms. After fixation, participants were shown a single image and had to identify it as a blast or non-blast cell. Participants received trial-by-trial feedback about their choices in this block.
The main task consisted of six blocks with 100 trials in each block. The main task was the same as the practice block where participants were ask to identify single images.
However, participants did not receive trial-by-trial feedback about their choices. They received feedback about their performance at the end of each block. The 100 trials in each block were composed of equal numbers of easy blast images, hard blast images, easy non-blast images, and hard non-blast images, fully randomized.
There were three manipulations across blocks: accuracy, speed, and bias. In the accuracy blocks, participants were instructed to respond as accurately as possible and were given 5 seconds to respond. In the speed block, participants were instructed to respond quickly and were given 1 second to respond. If they responded after the deadline, they received the message "Too Slow!" In the bias blocks, participants were shown a probabilistic cue on half of the trials. The cue was a red dot that was shown after fixation for 500 ms. The cue identified the upcoming image as most likely being a blast cell. The cue was valid 65% of the time. Participants were instructed about the validity of the cue at the start of the block. The order of the first three blocks was randomized, but with the constraint that there was one block for each type of manipulation (i.e., accuracy, speed, and bias). The order of the last three blocks was identical to the order of the first three blocks.
After completing the main task, participants completed a version of the Novel Object Memory Test (NOMT) (Richler et al., 2017). The NOMT is modeled after the Cambridge Face Memory Test (Duchaine & Nakayama, 2006) and provides a measure of general object recognition ability. In our experiment, we used two categories of novel objects (Ziggerins shown in Fig 1e and Greebles shown in Fig 1f). For each category, participants started with a learning phase where a target object was shown in three views followed by three test trials where the target was shown alongside two distractor objects.
Participants received trial-by-trial feedback during these trials. This learning procedure was repeated for 6 target objects (the six targets for each category are shown in Fig 1e   and 1f). Each target object was slightly different from the other targets in the same category. Following the learning phase, participants completed 54 test phase trials where they had to select which of three objects was any one of the six studied targets.

Modeling Methods
To gain insight into participants decision process beyond what is possible with statistical analysis of behavioral results alone, we use the canonical Diffusion Decision Model (DDM) of speeded decision making. The DDM posits that over the course of a decision, evidence stochastically accumulates over time until a confidence threshold is reached and a decision is initiated. This model has three core elements that makes it valuable in assessing participants behavior on the blast identification task. 1) How fast people accumulate evidence over time is encoded in an accumulation rate parameter (d).
A high / low rate indicates better / worse performance on the task. 2) The level of confidence a person requires to make a decision (i.e. level of caution) is encoded in a threshold parameter (a). 3) Finally, any initial preference for responding one way or the other prior to seeing the stimulus is described by a bias parameter (z). These are the three critical characteristics / parameters in the model that we will rely on to make inferences.
See Figure 2 for a schematic description of the DDM.
The full version of the DDM that we use here is comprised of 9 (or 10) free parameters: accumulation rates for easy and hard blast images (d BE , d BH ), accumulation rate for easy and hard non-blast images (d NBE , d NBH ), trial to trial variability in those accumulation rates (s d ), start point (z, which determines the initial bias), trial to trial variability in the start point (s z ), evidence threshold (a), encoding and response time (t ND ).
There is also a parameter encoding within trial stochasticity (s). However as is common, we fix this parameter to a value s=0.1 to avoid parameter degeneracy in the model (one parameter must be fixed). For the cueing instruction data, we introduce an additional parameter to denote the bias on trials where the cue is actually shown (z cue ). This will allow us to determine if the cue has any discernible effect on initial bias. Given that the speed, accuracy, and cueing instruction conditions all have the potential to influence people's behavior in different ways, we fit each instruction condition separately and do not assume up front that any model parameters are the same across experimental conditions.
We use a hierarchal Bayesian algorithm to fit DDM to all participants' data simultaneously, providing an account of the choices made and the full distribution of response times at both the individual and population levels. Given the high level of correlation between model parameters in this model, we utilize Differential Evolution Markov Chain Monte Carlo (DEMCMC) (Turner & Sederberg, 2012) to carry out this Bayesian estimation. Since the DDM does not have an analytically tractable closed form likelihood function, we utilize a recently developed approximation, the Probability Density Approximation (PDA) method (Holmes, 2015;Holmes & Trueblood, 2017;Turner & Sederberg, 2014), to approximate the likelihood of each parameter set sampled.

Behavioral Results and Discussion
We first examined average accuracy on the 60 practice trials preceding the main blast identification task to see how well participants learned to identify the images. The proportion of trials answered correctly in the practice block was 0.73 (SD = 0.09). We removed three participants with accuracy less than two standard deviations below the average because these participants were likely not engaged in the task.
We compared the proportion of correct responses in the accuracy and speed blocks for the four image types using a Bayesian Repeated Measures ANOVA. For this analysis, we compared five models: (1) a null model, (2)  (and not their interaction) was preferred to all other models (BF Model = 11.39) as well as to the null model (BF 10 > 1,000). We also calculated the BFs for the inclusion of specific variables (BF inclusion ). The BF for inclusion of image type was large, BF inclusion > 1,000, but the BF for inclusion of block was much smaller, BF inclusion = 2.39. This suggests that participants' ability to correctly identify images was more strongly influenced by image type than speed versus accuracy instructions. Figure 3a shows the proportion of correct choices for speed and accuracy blocks for the four image types. As can be seen in the figure, participants made more correct identifications for the easy images as compared to the hard images. Performance was the worst for the hard non-blast images.
We used a similar analysis to compare the mean response times in the accuracy and speed blocks for the four image types. The model including both main effects (block and image type) was preferred to all other models (BF Model = 2.83) as well as to the null model (BF 10 > 1,000). The BF for inclusion of block was large, BF inclusion > 1,000, but the BF for inclusion of image type was much smaller, BF inclusion = 1.41. This suggests that accuracy versus speed instructions had the largest effect on mean response times. Figure   3c shows the mean response times for speed and accuracy blocks for the four image types. As illustrated in the figure, participants responded more quickly in the speed blocks than the accuracy blocks.
Next we examined the proportion of correct responses in the bias condition when the cue was present versus absent for the four image types using a Bayesian repeated Measures ANOVA. Similar to the above analyses, we compared five models: (1)  as to the null model (BF 10 > 1,000). The BF for inclusion of image type was large, BF inclusion > 1,000. The BF for inclusion of cue presence was 0.11, suggesting that the presence / absence of the cue had no impact on choice behavior. Figure 3b shows the proportion of correct choices for the bias blocks when the cue was present versus absent for the four image types. Similar to the accuracy and speed blocks, participants made more correct identifications for the easy images as compared to the hard images.
Performance was the worst for the hard non-blast images.
We then compared the mean response times in the bias condition when the cue was present versus absent for the four image types. The model including only the main effect of image type was preferred to all other models (BF Model = 4.15) as well as to the null model (BF 10 > 1,000). The BF for inclusion of image type was large, BF inclusion > 1,000. The BF for inclusion of cue presence was 0.64, suggesting that the presence / absence of the cue had no impact on mean response time. Figure 3d shows the mean response times for the bias blocks when the cue was present versus absent for the four image types. As can be seen in the figure, participants responded more quickly for the easy non-blast images as compared to the other image types.
Next, we compared identification performance on the main task with performance on the NOMT. The average proportion of correct responses on the NOMT was 0.71 (SD = 0.10). We correlated NOMT performance with performance on the identification task for easy and hard trials separately. As shown in Figure 4a, we found a positive correlation between NOMT performance and performance on the easy trials of the main task (r = 0.39, BF10 = 2.57). As illustrated in Figure 4b, there was no correlation between NOMT performance and performance on the hard trials (r = 0.12, BF10 = 0.26).

Modeling Results and Discussion
We fit the DDM to each of the speed, accuracy, and cueing instruction conditions separately. It is in principle possible to fit the totality of the data at once as is often done.
Typically this is accomplished by fixing certain parameters (accumulation rates for example) to be the same across instruction conditions while others (threshold for example) are condition dependent. This however restricts up front the properties of the model that can vary between conditions. By fitting the three conditions separately, we allow maximal model flexibility, so that the data can determine what is the same or different across conditions.
To determine if the model was able to capture the data (which is necessary for it to be useful), we 1) extracted the mean parameters for fits to each of the conditions, 2) calculated the predicted choice proportion and mean response time (RT) for each condition, and 3) compared those predictions to choice proportions and mean RT's from data. Results ( Figure 5) show that the model provides a good accounting of most aspects of the observed data. In each of these figure panels, the diagonal line represents the line of perfect agreement (prediction = observation), with results lying close to this line in most cases. There are two exceptions. First, there is more spread in the fits for the cuing condition. Note however that each of the cueing experimental conditions has ½ as many observations as the speed and accuracy conditions (due to the presence of cue / no cue trials). Thus increased noise in the data would be expected. Second, the model does not provide a good accounting of the long response times, which is a common issue with DDM and other similar models since it is a model of speeded decision making. Overall, the model fits the observed data well and thus we will further analyze the model results.
We next look at the best-fit values for the three key model parameters linked to behavioral characteristics of interest here (accumulation rate, threshold, bias). Figure 6 shows posterior distributions (best fit parameter distributions) for these parameters in each of the three instruction conditions. Analysis of the drift rate estimates shows a consistent pattern across conditions. As expected, evidence accumulation rates are lower for more difficult images, regardless of instruction condition and difficulty. There is however a significant difference between participants' ability to perceive the characteristics of blast and non-blast images respectively, as evidenced by the fact that d NB ≠ d B . It appears that the characteristics of hard non-blast images are the most difficult to discern while the characteristics of easy non-blast are the simplest. This is consistent with behavior observations (Figures 3 a,c) where participants made more errors on hard non-blast images. Further, while participants had similar numbers of correct and error responses on easy blast and non-blast images, they responded more quickly in the easy non-blast case (in the accuracy condition only).
Results additionally show that there is no detectable bias in the speed or accuracy conditions. That is, participants had no implicit preference for identifying cells as either blast or non-blast. The introduction of a cue indicating a higher likelihood that the subsequent image is a blast does appear to introduce some bias. While this bias is relatively small (~5% of the threshold), it did appear to reduce the number of errors in cued easy, blast trials ( Figure 3b). Comparison of the threshold parameters between the speed and accuracy conditions suggests that the speed instruction dominantly influences the threshold parameter. Thus, under speed instructions, it appears that participants become less cautious rather than more perceptive or focused (which would show up in rates).
Next, we compared participants' performance on the NOMT with measures of speed, accuracy, and bias derived from this modeling. To do so, we used Bayesian linear regression to predict NOMT performance using the best-fit parameters from the DDM for each individual. We carried out the linear regression analyses separately for accuracy, speed, and bias blocks since the model was fit separately to these conditions. For the accuracy and speed conditions, there were 7 predictors (t ND, d BE , d BH, d NBE , d NBH , a, bias (a-z/2)). For the bias condition, there were 8 predictors since there were two different biases in the model (one for cued trials and one for uncued trials). We examined all possible combinations of predictors (2 7 =128 models were fit for the speed and accuracy conditions and 2 8 =256 models were fit for the bias condition). For the accuracy condition, the preferred model was the one with only d NBE and bias (BF Model = 22.38 and BF 10 = 225.44). For the speed condition, the preferred model was the one with only d NBE (BF Model = 10.86 and BF 10 = 188.07). For the bias condition, the preferred model was one with all drift rates (d BE , d BH, d NBE , d NBH ) and both bias parameters (BF Model = 27.09 and BF 10 = 346.26). For each condition, we also calculated the BF inclusion of each predictor as shown in Table 1. As can be seen in the table, the BF inclusion for the d NBE parameter was large for all three conditions. In particular, participants with larger d NBE parameter values had better performance on the NOMT. For the accuracy and speed conditions, the BF inclusion for all other parameters was small (less than 3, which is typically considered 'anecdotal'). For the bias condition, the BF inclusion was small for all parameters except the bias parameter for uncued trials. In particular, better NOMT performance was associated with smaller bias parameter values. These results show that the primary cognitive DDM parameter that correlates with NOMT performance is the evidence accumulation rates on easy non-blast images. This suggests that a persons' general object recognition ability transfers (to some extent) to perception of easy images, but not hard images. We posit a possible explanation for this limited correlation. There are two basic levels of information contained within these images associated with 1) gross morphological characteristics (which are more readily learned) and 2) more detailed characteristics (which require more practice and expertise).
Easy blast and non-blast images could potentially be distinguished primarily based on gross morphology while discrimination of hard blast and non-blast images requires consideration of more detailed characteristics. In this case, superior general object recognition capabilities in this untrained population appears be associated with improved ability to recognize those gross characteristics, but does not provide any significant advantage where consideration of fine details (and hence expertise) are required.

Conclusions
In this study, we took a joint experimental and modeling approach to investigate the cognitive processes involved in cancer cell image detection in diagnostic pathology.
While this study was performed with non-expert undergraduate students, we used clinical images curated by a panel of expert pathologists. Despite the lack of experience with medical image analysis, this pool of participants was able to learn the characteristics of blast (abnormal) cells and to discriminate between them and normal cells with a reasonable level of accuracy.
Using standard paradigms in perceptual decision-making (e.g., speed-accuracy tradeoff), we found that time pressure led to a reduction in caution in participants, with a commensurate increase in errors, as is common in other domains (Bogacz, Wagenmakers, Forstmann, & Nieuwenhuis, 2010;Starns & Ratcliff, 2010;Wickelgren, 1977).
Additionally, we found that participants had no implicit bias to classify images as either blast or non-blast. Even an external cue, indicating an increased likelihood of the subsequent image being a blast cell, only introduced a slight bias. Finally, we analyzed participants' ability to discriminate blast and non-blast images of different difficulty by showing images that were deemed (by expert pathologists) to be more or less difficult to identify. Unsurprisingly, participants had more trouble with difficult images. Further analysis of both behavioral and modeling results however revealed that participants had different capacities to discriminate blast and non-blast cells, as indicated by a robust ordering of evidence accumulation rates across instruction conditions. This is possibly due to the fact that there is one blast cell type while there are multiple non-blast cell types (all other white blood cell types).
In addition to considering performance on this task, we investigated how general object recognition ability impacts error rates using the NOMT. Results show that there is a positive correlation between performances on the two tasks. That correlation is however limited to performance on easy trials and the only cognitive model parameter to correlate with NOMT performance (across all three instructional manipulations) is the rate of evidence accumulation on easy non-blast trials. We hypothesize this limited correlation is the result of participants with good object recognition ability being able to better learn and recognize gross morphological characteristics of these cell types, but not more detailed characteristics, which require further training and expertise.
In conclusion, we find that differences in the perception of different cell types (which is of course task specific), caution, and general ability to learn and recognize objects all influence performance, though to differing extents. Further, and somewhat surprisingly, this participant population exhibits essentially no bias in their decisions, and probabilistic cues intended to influence that bias appear to have little effect on it. While experimentation alone can be used to quantify and describe errors, the joint experimental and modeling approach used here provides a way to focus more closely on the cognitive factors that influence decisions. Hopefully this study and future ones will help shed light on the role of cognitive processes in medical image perception and decision-making.    Horizontal axes indicate response accuracy on the NOMT and vertical axes response accuracy on the identification task. Lines represent the best fit regression lines.
Correlations computed separately for the easy (a) and hard (b) image classes. Within each image class, the proportion of correct responses is calculated over all instruction conditions.   d) e) f)