{'fraction_non_alphanumeric': 0.023619869454585683, 'fraction_numerical': 0.010012169487775197, 'mean_word_length': 4.858392741412832, 'pattern_counts': {'":': 0, '<': 2, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Modeling Human Visual Detection Using Deep Networks

The work in this report describes the use of machine learning to model human visual detection. This is in contrast to typical machine learning models, which seek to optimize detection performance overall, e.g., precision versus recall or F1 scores. Instead the goal is to develop models that can accurately match humans’ abilities to detect objects in images. Modeling human performance in tasks has long been a benchmark for artificial intelligence (AI), from computerized chess to conversational robots. Until recently, the goal of AI has been to match human performance in complex tasks, with the assumption that human performance is an attainable goal. But there are many AI algorithms that have far surpassed humans in, for example, object detection in large image databases or games such as Go. What is different about this work is that the objective is to accurately model humans’ performance in visual detection tasks, with the supporting task of knowledge discovery on how humans interpret complex images to detect objects. To accomplish this, deep learning architectures designed for image classification are adapted, extending these architectures to predict detection statistics of human observers. Results comparing human performance against machine performance indicate a strong correlation of the human and machine. Furthermore, explainable AI (XAI) methods demonstrate that the machine learning algorithms are interpreting the images in intuitive ways, which supports the notion that the algorithms are learning an accurate representation of how humans interpret


Chapter 1 Introduction & Goals
Neural networks have become progressively more effective in their various domains.
Many applications of machine learning and neural networks aim to exceed human capabilities in things like identifying cancerous cells [1], but in this study my objective is trying to match, rather than exceed, human performance in an observation task.
That is, given a set of data that were presented to human volunteers, can the network learn which images had objects that most volunteers were able to identify, and can the network estimate how many were able to identify it? A fair amount of work has already been done on looking at various aspects of human visual perception. For instance, Yuan and Li [2] developed a testing method and a model to test various user interface (UI) designs and predict how long it would take users to identify a particular object amongst clutter, which would likely be of great interest to websites that generate revenue via advertisements. Another study published by Dodge and Karam [3] compared how well humans were able to correctly classify noisy images against how well a trained network did with those same images. These are very interesting topics, but they do not look at the first of the three main focuses this project addresses, which is that of predicting the probability that a human will be able to see a given object. Development of a model that accurately mimics a large base of volunteers will greatly decrease the cost of obtaining similar probability estimates for future images (by limiting the number of necessary human volunteers), and will do so at much greater speed.
The second problem this project looks at is ordinal classification. Ordinal classification has received a fair amount of attention in literature. Multi-view max pooling [4] has been introduced as a technique to increase data augmentation capabilities and improve network performance on ordinal tasks with respect to images. For instance, improving the classification of the relative age (young, middle-aged, old) of a person in an image. Frank et al. [5] also looked at the task of ordinal classification, but took the route of training (C − 1) classifiers, where C represents the number of classes, effectively making the problem one-versus-all. Perhaps the most relevant publication, by Cardoso et al. [6], describes the the most similar approach to that explored in this report, which is that of considering predicted classes farther away from the correct class as worse predictions than those closer to the correct class. Unfortunately, [6] focuses more on calculating the error of poorly ordered class predictions. In the problem domain faced in this project, the network is not predicting a class order, so the problem becomes evaluating error where classes have some inherent numerical value such that, given an image and an observed Probability of Detection (P d ), e.g., 0.0, a prediction vector representing 0.1 produces a more optimal loss value than that of a prediction vector representing 0.5.
The third and final area of focus is incorporation of explainable AI (XAI). There are several legends about networks that achieved incredible training accuracy only to be completely thwarted by the testing set due to some minor difference the network engineers never considered. Incorporating XAI provides evidence that the network is not training on irrelevant image features.
Some of the material in this report was published in [7].

Data
The data used for this project consist of renders of an environment and an object within that environment, which varied across the horizontal-axis but with a fixed vertical coordinate at half the image height. The image is then mirrored horizontally, and these two images are then presented to a human observer for a short time (∼ 2 seconds). The volunteer must decide if an object exists in one of the two images, and if so, respond by pressing a button indicating which side they believe the object to be on. The probability of detection (P d ) of an object can then be calculated by dividing the total number of true positives by the sum of true positives and false negatives. The aggregated P d of the human subjects is used as the truth data for this project, while the corresponding images with an object in them are used for the prediction input.

Network Architecture
A convolutional neural network (CNN) was a natural solution to solve the problem that involved training on images. A great deal of work has already been done in optimizing network architectures for image recognition problems, and so a transfer learning approach was used to compare models that leveraged Xception [8] and ResNet50 [9]. However, the high resolution of the input images, compared to the sizes of the training sets for which Xception and ResNet50 were designed, necessitated the addition of extra input layers to "down-sample" the image; albeit the down-sampling function is a learned attribute of the network. ResNet50 accepts 224 × 224 images as input, and Xception accepts 299 × 299 images, so the training image from our study was zero-padded to a square shape of 1196 × 1196 for Xception or 1120 × 1120 for ResNet50, and then down-sampled using a two-dimensional convolutional layer with 3 filters to match each network's native input size. Furthermore, the different problem domain that Xception and ResNet50 were intended for required a change in the classification layers of these networks. Both Xception and ResNet50 were designed Two approaches to predicting the P d were explored. The first such paradigm involved one output neuron activated by a sigmoid function. The second paradigm was Thus, such a design trains on two inputs, one full resolution input, and that same full resolution input cropped to a pretrained network's native size centered around the object, based off the assumption that a perfect object detector would be able to provide such an input. This two-input architecture has been dubbed the parallel path architecture. This gives a total of 8 different network architectures that are compared in Chapter 3. These architectures are referred to as X-to-Y networks, where X indicates the number of inputs, with 2 being the parallel path architecture, and

Ordinal Classification
The idea behind ordinal classification is to get a measure of the level of confidence of the network. The way this works is by using a one-hot encoded vector instead of a single continuous value. categorical crossentropy produces g(y,ŷ 1 ) = g(y,ŷ 2 ) ≈ 16.12 (where g(·) represents categorical crossentropy). The problem with existing loss functions, such as those discussed here, is that the error is calculated based on whether a class is predicted correctly or not, and pays no regard to how close the predicted class is to the true class. In traditional classification tasks, this approach makes sense. There is no ordering relation between classes that represent a boat and a pomegranate. Consequently, the network is either correct or incorrect, and there is no degree of incorrectness to consider. However, such an ordering relation does exist in the ordinal classification task discussed here. To resolve this issue, a new loss function called Squared Ordinal Loss is introduced, which is defined as where y is a one-hot encoded vector (rounded to the nearest percentage point) indicating what the observed P d is in terms of classes, andŷ is the network's prediction.
C is a K ×K weight matrix that allows for the ordering relation between classes to be accounted for by linearly applying more weight to classes farther away from the true class according to C ij = |i − y j |, though one could imagine using any weight mapping appropriate for their problem. Multiplying C by y retrieves the correct weight vector from C and applies those weights to the prediction. Summing the squares of the weighted predictions provides a loss for the training sample. Considering the above example, where y = [1, 0, 0, 0], the weight matrix is and multiplying by y reduces the matrix into the appropriate weight vector C 0 = [0, 1, 2, 3]. Thus, a prediction of y 1 should be calculated to be much worse than a prediction of y 2 . Indeed, calculating these losses shows that L ORD (y 1 ) = 0+0+0+3 = 3 > L ORD (y 2 ) = 0 + 1 + 0 + 0 = 1. A sample weight vector extracted from the weight classes continue to grow more incorrect to a maximum weight of 60 for the 101st class, which corresponds to a P d of 100%. Now that the ordinal loss function is defined, the application of an XAI approach to provide interpretability of the trained deep network will be discussed next.

Class Activation Mapping
Class activation mapping (CAM) [10] is an XAI technique that can be used to visualize the pixels that most impact a network's prediction. Shown in Figure    This technique was applied to the data using a 1-to-1 Xception-based architecture, and in the majority of cases, CAM produced a circular heatmap centered on the object which Figure 2.11 demonstrates. The instances in which it did not were images where the object was nearly invisible, coinciding with a P d near 0.0, which Figure 2.12 demonstrates. The difference between these two high-and low-P d heatmaps is rather stark. The network is very tightly focused on one area of the image that has a P d of 1.0, and is not particularly focused at all in the image with a low corresponding P d .
Finally, to visualize the pixels that tended to be weighted most highly by the network, an input image where all pixels had a value of 1 was used. This CAM heatmap is depicted in Figure 2.9, and shows that the network weights pixels towards the side of the input more heavily. A higher-resolution version shown in 2.10 upsampled to 1196 × 1196 via bi-linear interpolation shows a smoother distribution of the weights.
The distributions shown agree with the majority of the object positions, which tended to be away from the center of the image; see Figure 2.13 for a histogram of the object locations in the image data set.

Results and Discussion
As mentioned in Section 2.2 there are 8 different network architectures being compared here, each with different base networks, different numbers of inputs, and different methods of classification being used. Each combination of these variables produce one architecture, which was trained and tested using 5-fold cross validation for 80 epochs, and with a decaying learning rate that started at 0.01 and decayed by an order of magnitude every 20 epochs to a minimum of 1 × 10 −5 at the start of epoch 60. These parameters, including the order and contents of each fold, were held constant across each trial to help ensure consistency in the results. In addition to comparing the architectures instantiated with the aforementioned variables, various weight initialization strategies are also analyzed in an effort to determine how such strategies affect the performance of the parallel architectures. An interesting note is that peak performance may include a prediction that indicates maximum uncertainty, i.e. a P d ≈ 0.5. The truth data in this study are calculated from the mean of human performance, which will never approach the same surety as truth data that, for instance, classifies an image as a cat or a dog. Because the probability of detection can vary depending on where the observer happens to be looking, eye or mental fatigue and the existence of outliers, among other factors, a perfect prediction will always be rare, so some measure of distance between prediction and truth must be used in every metric.  The experiment consisted of setting up a trial for each network architecture and different methods of instantiating the weights of those that are parallel. Each trial consisted of training a network 5 times, with each training instance holding back a different fold to use as a validation set. These folds were determined by splitting up a randomized collection of references to the data into five unique and disjoint sets.
These sets were then serialized so that the same data in the same order could be used across the different architectures. Each architecture was trained for 80 epochs on each of the five folds, and the mean and standard deviation of the MSEs from the Three methods of initialization were explored for the parallel architectures. The first, random, simply randomly initialized all weights. The second, imagenet, loaded each subnetwork of the parallel architecture with the pretrained imagenet weights for that network. Finally, pretrained refers to a weight initialization that used the weights of the matching, fully trained non-parallel network. The comparison among these results is made in Table 3.3. For brevity, only the arithmetic mean of the validation loss across the five folds is shown.
To compare the results across the different architectures, an unpaired two-sample T-test is used between the sample of means computed over the folds for the architectures being compared. The T-test shows the statistical significance of the possible different performance, highlighting which network (and weight initialization variant , the lower bound of the CI will be greater than 0. If mu 1 < µ 2 (where < represents statistically lesser) the upper bound of the CI will be less than 0. If µ 1 == µ 2 (where == represents statistically equivalent), 0 will fall between the CI's upper and lower bounds. The cells which show either that a parallel design outperformed the single-input design or that ordinal loss outperformed MSE are green.
Those that show the opposite are highlighted in red. In the instance that there is no difference, the text will be black.
The first comparisons will be between the various weight initialization strategies explored. The results shown in Tables 3.6 and 3.7 demonstrate that for most pairings there was no significant difference in mean performance between any pair of initialization methods. This somewhat surprising as the pretrained initialization strategies effectively received more training time, via each half of the parallel network getting instantiated from weights obtained from a non-parallel network already trained for 80 epochs. In general, it appears that the weight initialization strategy does not greatly affect the loss. The only situation in which an improvement is seen is in the parallel ResNet50 regression architecture, where randomized initialization was outperformed by alternative strategies. Given these results, it appears that weight initialization does not play an important role, and the remaining architectural comparisons will take place between networks instantiated with imagenet.
Tables 3.8 and 3.9 show the comparisons between the regression and ordinal networks, and between the parallel and non-parallel architectures respectively. Table   3.8 reveals that for all cases except one, ordinal prediction and training results in

Chapter 4 Conclusion and Future Work
In this report, existing networks optimized for image classification tasks are leveraged in a transfer learning approach in order to predict human visual detection performance. Utilizing a weight visualization technique called CAM rationalized network performance and indicated the networks behave as expected and as desired. Experimenting with various weight initialization strategies showed that how weights are initialized is irrelevant, and testing the remaining permutations of variables against each other to find the most optimal network revealed that parallel instances of the Xception architecture utilizing the ordinal loss function outperformed the alternatives. Finally, and most significantly, a new loss function is explored that attempts to classify an image's probability of detection as one of 101 discrete ordered values representing discrete bins of percentage points, allowing for a single prediction to simultaneously provide information about the appropriate class and a measure of confidence in that prediction.
In future work, exploring how scaling the number of parallel networks impacts performance may yield favorable results, given that the parallel architectures generally outperformed the non-parallel architectures. Further investigation into how the distribution of classified P d "bins" can inform epistemic uncertainty, and the feasibility of transforming that hyperparameter into a variable that can be learned may also lead to interesting finds.