{'fraction_non_alphanumeric': 0.029105863094643962, 'fraction_numerical': 0.0079252590182261, 'mean_word_length': 4.88787023977433, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 13, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Embracing New Techniques in Deep Learning for Estimating Image Memorability

Various work has suggested that the memorability of an image is consistent across people, and thus can be treated as an intrinsic property of an image. Using computer vision models, we can make specific predictions about what people will remember or forget. While older work has used now-outdated deep learning architectures to predict image memorability, innovations in the field have given us new techniques to apply to this problem. Here, we propose and evaluate five alternative deep learning models which exploit developments in the field from the last five years, largely the introduction of residual neural networks, which are intended to allow the model to use semantic information in the memorability estimation process. These new models were tested against the prior state of the art with a combined dataset built to optimize both within-category and across-category predictions. Our findings suggest that the key prior memorability network had overstated its generalizability and was overfit on its training set. Our new models outperform this prior model, leading us to conclude that Residual Networks outperform simpler convolutional neural networks in memorability regression. We make our new state-of-the-art model readily available to the research community, allowing memory researchers to make predictions about memorability on a wider range of images.


Introduction
A person's ability to remember an image is thought of as something that varies from person to person. Some people have "good" memories, while others forget images easily. While this is certainly true, recent research has found a surprising consistency across individuals in what they remember and forget, driven by the images themselves; images can be intrinsically memorable and forgettable across observers. Because of this high consistency in memory performance, each image can be assigned a memorability score between 0 and 1 [1,2]. These "memorability scores" are determined experimentally by testing how well a large group of people remember an image. This is an expensive and time-consuming process, especially at scale. There are also still big open questions about what determines the memorability of an image. In this paper, we propose a new method for evaluating the memorability of images, and reveal an important contribution of semantic information in memory performance.
Memorability scores that can be assigned to an image are conventionally generated using a memory test. A participant is given a series of images and is asked whether or not they have seen them before. By aggregating how human participants score on each image, researchers can generate an estimate for the ground truth memorability of that image. Memorability scores generated from a subset of the participants are a good estimator of the memorability scores from the complement of that subset (measured by a high Spearman rank correlation between them) [3,4]. The question then is how do these memorability scores arise, and why are they so consistent across people?
As soon as one views an image, high level visual and memory areas in the brain show a sensitivity to the memorability of that image, regardless of task or whether participants consciously remember that image or not [5][6][7]. This sensitivity may even occur in the brains of non-human primates, suggesting that memorability is evoked by images themselves, during late perceptual processes [8]. A current hypothesis is that memorability could reflect how perceptual inputs are prioritized for memory, given that not all perceived information can be remembered. Specifically, memorable items may be those that we visit first during retrieval in the larger search space of memories, and memorable items are reinstated earlier in the brain than forgettable ones [9]. However, the stimulus features that cause an image to be memorable are still under active exploration. Memorability shows low correspondences to low-level visual features like colors and contrasts, and is also not predictable by comprehensive sets of human-labeled features like attractiveness or interest. Computational models promise to be the next step in understanding what causes a given image to be remembered [10].
Given that memorability is intrinsic to an image, then this implies that we should be able to derive memorability directly from an image. The earliest work in applying computer vision to memorability used hand-engineered Histograms of Oriented Gradients (HOG), and dense Scale Invariant Feature Transforms (dense-SIFT) [11]. While these methods can be very effective, they are hand-engineered, meaning that they need to be tuned by a human, instead of being determined algorithmically, and thus require assumptions about the relevant visual features. As end-to-end trained vision techniques like convolutional neural networks (CNNs) became available, the problems associated with HOGs and dense-SIFT models could be avoided, giving us better performing estimators at the cost of interpretability.
In recent years, other methods for estimating memorability have been proposed using convolutional neural network regressions. The convolutional neural network is a deep learning technique that scans over an image in small regions, and optimizes the process of analyzing the regions, rather than optimizing over every data dimension (every pixel and channel). This generally yields good performance while lowering training costs and preventing vanishing gradients. The most well-known of these is MemNet [12], although it does have a few challengers which have employed attention models [13], image captioning features [14], and transfer learning [15]. MemNet's memorability estimations had a rank correlation of 0.57 with held-out ground truth scores from its training dataset and is the most commonly used neural network regression for this purpose. However, some roadblocks have emerged for researchers trying to use MemNet. MemNet was built in Caffe, a deep learning framework which has been defunct since shortly after MemNet's publication, and its online demo is no longer functional. Importantly, MemNet uses techniques which are no longer preferred for computer vision regression. Techniques like residual neural networks, semantic segmentation, and new optimization methods have been developed and become widespread in the meantime, and have improved the ability of these networks to derive richer, semantic information from an image.
Here, we present a new state-of-the-art model for predicting image memorability based on recent advances in computer vision and our understanding of memorability itself. Using these new techniques, as well as an improved combination dataset, this new model can be more generalizable to the image-space. Further, these new advances reveal an important role of semantic information in image memorability, and have spurred the creation of a new dataset for examining the role of different features in image memorability.

Methods
First, we will reproduce MemNet on a modern framework, and explore its statistical properties in a controlled environment. Then, we will outline deep learning methods that utilize new techniques for the same problem, and show that their use results in large performance increases. The techniques we will discuss are based around residual neural networks (ResNets), a specialization of convolutional neural networks. Next, we will explore these new models, and show that the addition of a residual neural network gives the model access to higher level conceptual features. Finally, we will show that our model makes successful predictions of memory across a diverse range of image sets, making it a useful tool for the research community.

Data Sources
The original MemNet was trained on a dataset called LaMem [12], the Large Scale Memorability Dataset. LaMem contains 58,741 images, all labeled with the average memorability for each image. If one splits all of the human participants into two groups, and takes the Spearman rank correlation between each group's average memory performance for a given image, the correlation will be 0.67, suggesting that people are indeed highly consistent in the images they remember and forget. The images in LaMem are a compilation of a few more well-known datasets, the MIR Flickr Image Set [16], the AVA (Aesthetic Visual Analysis image set) [17], the Affective Image Set [18], two saliency image sets [19], [20], the Scene Understanding (SUN) dataset [21], an image popularity dataset constructed using Flickr [22], an anomaly detection dataset [23], and a dataset designed for attribute-based object description [24]. These datasets were chosen because they have other metadata associated with them, like popularity on Flickr, or eye tracking data, or aesthetic ratings. All together, LaMem has the benefit of being relatively naturalistic and diverse. The dataset was compiled with the intention of looking for correlations between memorability and these other data points. LaMem is a well suited dataset for this task because of its size and relative quality.
Our reimplementation of MemNet, as well as all of our models, were trained on a mixture of LaMem and MemCat [25]. MemCat is a smaller, but higher quality dataset of images and memorability scores. MemCat was designed to help understand how memorability changes across and within categories of images. It has 10,000 images, and its inter-human rank correlation is 0.78. This value generally increases with the amount of samples the dataset has per image, and it scales with the certainty of each value. Taken together, this implies that the memorability scores in MemCat are better estimations of human performance.
These two datasets have some key features that can influence the behavior of our deep learning models. LaMem was compiled from a group of other datasets, which in turn were compiled to study secondary metadata of that image, like popularity on Flickr, aesthetic rating, etc. Considering the data sources for LaMem, we can notice that they are not a general sampling of naturalistic images. Flickr, for example, is an online community focused around sharing interesting images, thus, datasets from Flickr have a selection bias toward less naturalistic images. Among the other data sources, we have images that are designed to trigger eye movement, images that are designed to be anomalous, images that are designed to be affective, and so on. For example, many of the images from Flickr are intended as artistic renderings, and not necessarily true reflections of reality. On the other hand, LaMem is very generalizable across image types. LaMem contains images of scenes, objects, people, and even a few Example LaMem Images Example MemCat Images cat cat forest (broadleaf) forest (broadleaf) Figure 1: Sample images from our combined dataset. The first row is from LaMem [12], and the second row is from MemCat [25]. The specific category is written underneath each MemCat image.
images of text and iconography ( fig. 1). The vast majority of LaMem's images are scenes, but the fact that it contains images of other types will improve the generalizability of the model across those types.
MemCat, on the other hand, was designed to describe several categories of object-focused images. Each image fits neatly into a category, like "cat" or "snowscape" and as such, every image is an object image, where the objects are largely unoccluded, and generally presented in context, and serve as the focus of each image ( fig. 1). This contrasts with LaMem whose images are in a less natural context, and less focused on presenting a single object per image. However, what MemCat sacrifices in type-generalizability, it earns back in within-object generalizability; there are roughly a hundred images of deciduous forests and a hundred more of coniferous forests.
The final dataset used in the current study is a combination of these two datasets, with a mixture of objects, scenes, and miscellaneous images. The trends and features found in the larger but lower certainty dataset (LaMem) get reinforced by the smaller, more precise dataset (MemCat), resulting in a final model that is powerful and generalizable on multiple axes.

Architectures
We will compare three deep learning architectures, two of which are novel, that serve as the backbone for five models. First we will examine the MemNet architecture as downloaded in its original CaffeModel form. We also re-implemented its architecture using the more modern PyTorch [26] framework. Second, we will describe our new ResMem architecture, an architecture that uses residual neural networks as a secondary feature, and used here to create two models, called ResMem and ResMemRetrain. The third architecture we implement is M3M, which introduces a tertiary feature, semantic segmentation. Our new architectures also employ relatively simplified preprocessing steps, which avoid the assumption present in prior models that memorability is preserved across cropped and normalized versions of images.

MemNet
We tested two implementations of MemNet, one using its original Caffe model, and one re-implementing the architecture of MemNet using more modern approaches and best practices in deep learning. First, we tested MemNet's architecture in its original Caffe implementation [12]. The main innovation in MemNet was the use of convolutional neural networks (CNNs) for feature decomposition. Convolutional neural networks were first created in the 1980s [27], but did not see widespread use until the 2000s when GPUs became widespread, and a GPU implementation was written for CNNs [28]. Then, the final piece needed to use CNNs at the modern level was to implement backpropagation [29]. Backpropagation speeds up neural network training by taking the derivative of the network's output with respect to each of the parameters of the model, and uses that to decide which parameters to try in the next step. This innovation was shortly followed up by AlexNet [30], a model that used these techniques for image classification. MemNet is designed after AlexNet ( fig. 2), which consisted of 5 convolutional layers (some of which are pooled) followed by 3 fully connected layers. MemNet differs from AlexNet in the number of filters and channels in each layer and the number of neurons in the fully connected layers, but the overall architecture is the same. The intention in building MemNet was to leverage AlexNet's unparalleled efficiency (for the time) at extracting features from images.
The Caffe model of MemNet made publicly available could not be run from the online model alone [12]. To allow the model to function, we implemented additional steps based on discussions with prior users of the network. First, the input images were offset by a preset image, which is purported to be an image mean. Second, the images were subjected to a Ten Crop Transformation, where the images were cropped five times to a certain size, once with the crop aligned to each corner, and once aligned to the center of the image. The images were then mirrored, and the process was repeated to yield ten crops in total. These ten crops have their memorability scores estimated by the neural network, and their estimations are averaged to get one score. These two transformations are common, and were individually considered good practices in 2015, but are two separate approaches to the same problem. After this, the score was transformed using a standard scaling method. In the original MemNet model, the parameters for this scaling were hard-coded, and were not part of the training process. This hard-coded scaling is not considered best practice, and for this reason, we also constructed a model using a modern framework, employing best practices, while using an identical architecture to MemNet.
For this second implementation of MemNet, we reverse-engineered it using PyTorch [26], a popular framework for designing neural networks in Python. We chose PyTorch because it gives researchers a lot of direct control over the architecture of their models, and we believe that it will be well-maintained for the foreseeable future. The original Caffe version of MemNet was pretrained on only the LaMem dataset, but here we trained the PyTorch implementation of MemNet on our broader mixed dataset. This allows us to directly compare it to our new model. Further information about the process of hyperparameter tuning can be seen in the model training section of the Supplemental Information.

ResMem
Historically, the next big advancement in using neural networks for computer vision was residual neural networks. ResNets are built by taking convolutional layers and connecting them across levels with a "residual connection." The design builds on constructs observed in pyramidal cells in the cerebral cortex. In reality, this is implemented as adding the previous layer to the convolution at each step [31].
The advent of ResNets allowed for convolutional neural networks to be built deeper, with more layers, without the gradient vanishing. We utilized a pretrained ResNet from the torchvision model zoo [26], specifically, the whole model and not just the convolutional features, and added it as an input feature for our regression step. The model we used is called "ResNet-152" and was originally trained for image classification using ImageNet, an industry standard dataset for image classification [32]. ImageNet is a dataset of roughly 14 million images, and for the ResNet-152 training process they are classified into 1000 categories that are then predicted by ResNet-150. The process of using a model that was trained on a different but similar problem is called transfer learning, and is a staple technique of deep learning. This model consists of 152 layers of residual neural networks, followed by a short fully connected section that produces a 1000-length feature vector. Since the ResNet was previously trained on a semantic classification problem, this means that the ResNet features we are using are a semantic representation of the input image. We combine this ResNet feature with AlexNet features to create ResMem ( fig. 3).

Figure 3:
A skeleton diagram for ResMem. The input image is run through a deep residual neural network, and an AlexNet feature architecture, and the two features are combined, then run through a fully connected deep neural network.
We can use this structure to make two models. In one, the optimization algorithm does not have access to the ResNet-152 parameters during training. This means that the intermediate ResNet features are categorical data as the original was trained on ImageNet [32], a standard image classification dataset. In the other, which we call ResMemRetrain, we allow the ResNet component of the model to retrain for the memorability task. This allows the model to specialize the ResNet features for this task, but the model loses the assumption that the ResNet features are semantically in line with ImageNet. By allowing ResNet-152 to retrain, the model gains specificity at the expense of a much longer training time. Further information about the hyperparameter tuning of these models can be seen in the Supplemental Information.

M3M
Finally, we added a third feature to our model based on Semantic Segmentation ( fig. 4). Semantic segmentation is a common computer vision task where a model takes an image and classifies the object represented in each pixel. The segmentation model we are using is called fcn_ResNet-50, and also comes from the PyTorch model zoo [33]. This new model was built without any retraining of the two pretrained models, as semantic segmentation is a very high memory usage task, and would overload standard graphics cards.
The hypothesis is that by adding another semantic feature to the model, the overall accuracy of the model will improve. Where the classification model ResNet-152 returns a 1000-vector with estimated probabilities for each category, fcn_ResNet-50 returns an array the same size as the input image, but instead of having three channels (Red, green, and blue) it has 21 channels, with estimated probabilities for each category for each pixel. This is considerably more semantic information, and also contains coupled spatial-semantic information.

Figure 4:
A skeleton diagram for M3M. In addition to the ResNet and CNN, a Semantic Segmentation feature is also sent to the fully connected deep neural network. Since semantic segmentation produces very high dimensional outputs (on the order of one million dimensions), this feature must also be passed through a convolutional neural network to down-scale it.
Of important note, Semantic Segmentation outputs a data array with more channels than a normal image, so it cannot be reshaped directly into a standard linear layer. To solve this, we utilize a small convolutional neural network to process the segmentation into more simple convolutional features.

Network Feature Analysis
Each layer of a convolutional neural network or residual neural network performs feature extraction. They extract features in the output of the previous layer, and the first layer extracts features from the input image. The "features" are patterns that appear in the data, and when optimized, the model picks out the patterns that are most useful in solving the problem. Because these features resist precise definition, it is difficult to tell what the model is sensitive to; they can only be analyzed post facto. However, the consensus is that early layers are more perceptual and the later layers are more conceptual [34]. While fully mapping out the relationships in a deep neural network of this size is impossible, performing what feature analysis we can gives preliminary insight into the types of features important for predicting the memorability of an image.
We can visualize the important features in the model by optimizing the inputs to maximally trigger a certain feature in the model. By running this process on a sampling of layers and features, we can get a sense of the important neural network features in our model. When we apply a qualitative label to each feature, we can also see what images trigger which features, and get a better idea of how those features contribute to memorability in combination.

MemNet
We tested the publicly available version of MemNet, which yielded a Spearman rank correlation between predictions and human memory performance of 0.565. This is within expected limits of the 0.57 reported in the paper [12].
Beyond the rank correlation we can also examine other statistical properties of the predictions.
Specifically, we can plot the distribution of ground truth memorability scores and compare them to the distribution of estimations ( fig. 5a). An optimal model would produce the same distribution as the ground truths, and these visualizations can give us a deeper understanding of the model's statistical behavior. Note that passing this test does not show that the model is correct, since the distributions could be the same and have mismatched scores, giving a high loss and a low rank correlation. Failing the test, however, is sufficient to show that the model under-performs. A common issue with complex estimation models is regression towards the mean. This is when a model optimizes loss by predicting the mean value in the input data with some small variation.
If that small variation is correlated with the ground truth, then the model will achieve a good Spearman rank correlation, but a mediocre loss. The distribution of MemNet predictions shows a tendency for this regression towards the mean.
When trained on the combination dataset, with early stopping determined by rank correlation on a validation set, the PyTorch implementation of MemNet has a rank correlation 0.55 on a held out test set, and an MSE Loss of 0.012 (Table 1). This implies that, on average, the memorability scores are accurate to 0.11. The reverse engineered MemNet's prediction distribution is fairly close to ground truth. However, the distribution is dense around the mean, and has a hard cutoff on the high end ( fig. 5b).

ResMem
The ResMem model using the pre-existing ResNet-152 parameters had a rank correlation of 0.66 and a loss of 0.009 on the test set, which corresponds to accuracy within 0.092 on average. By looking at the prediction distribution for ResMem ( fig. 5c), we see that the shape is roughly correct for most of the range, but there are sharp cutoffs at the high and low end. We can infer that the model is highly accurate within the 0.6-0.9 range, although it struggles outside this region.  . 5d). While prediction success is cut-off at the low end at 0.411, overall this is much closer to the ground truth distribution. The reduced performance for low memorability images is likely due to the fact that only a few images have low memorability scores. The lowest score in the dataset is 0.2, and the lowest score that ResMemRetrain will assign is 0.411. We anticipate this is not an important issue since only 0.6% of images in the dataset have a memorability score below 0.411.

M3M
Testing with M3M yielded mixed results. While adding the semantic segmentation features to the model does increase rank correlations to 0.68, this increase is very small compared to the heavy increase in complexity. Additionally, MSE loss on the test set was 0.009, which is not meaningfully different from the MSE loss on ResMem and ResMemRetrain. This demonstrates that the semantic segmentation information does not contribute to memorability any more than the higher level semantic data from ResMem does. While this is a null result, it is a null result that can tell us about the state of memorability: the exact location of semantic objects in the image does not matter as much as the simple existence of the semantic objects.
The distribution plot for M3M displays similar behavior to ResMem ( fig. 5e). This also supports our earlier conclusion that M3M's performance increases in terms of rank correlation and loss are not worth the added costs. One training run of M3M took 30 hours to train on an Nvidia 1080TI. For comparison, ResMemRetrain takes 15 hours to run through the same amount of training steps, and ResMem without retraining and MemNet take 80 minutes.  Overall our new models outperform MemNet, while still maintaining reasonable train times (Table 1). Figure 6 shows the images for which our model makes the most successful predictions, and which ones have the largest prediction error. Adding a Residual Neural Network step to the model improves predictions by a considerable margin. Since ResMem employs both conceptual and perceptual features in its estimation process instead of just perceptual features, this implies that both types of features contribute to memorability. While the improvement on the general performance metrics shows that our multi-feature models outperform MemNet, it can also be useful to examine the other statistical properties of the model's estimations.

Model Evaluation
Given the improvement in the model made by the addition of the semantic features from ResNet, one hypothesis could be that image category information alone is sufficient for making successful predictions of memorability. To test this question, we used a ResNet pretrained for image category classification, and conducted a simple linear regression on the estimated category membership for predicting memorability. This model achieves a Spearman rank correlation of 0.477, while using AlexNet instead yields a rank correlation of 0.425. Thus, while just the image category alone can predict memorability to some degree, this basic model under-performs all models reported here that are designed specifically for predicting memorability.
Another interesting question is how well ResMemRetrain can predict the memorability within individual categories; are there some image categories that are better predicted than others? To test this, we evaluated the accuracy of ResMemRetrain across the categories in MemCat. This is presented with the caveat that MemCat is not large enough to have a sizable sample in every category in both the training and the test set, so the figures presented are calculated on the entire image set, and includes images present in both the training and test set. Thus, the performance measures should be considered as relative measures to compare across categories, rather than absolute measures of performance. The top 5 and bottom 5 subcategories defined by mean squared prediction error are presented in Table 2 and all superordinate Mem-Cat categories are presented in Table 3. In general, our model is best able to predict the memorability of landscape images in comparison to images of animals in terms of rank correlation. The full table of all subcategories is available on OSF.

Network Feature Analysis
After running the activation optimization method on the ResNet component of ResMem ( fig. 7), we visualized examples of features that drive some of the tensors in the different layers. In earlier layers, we observe more basic features, like edges and orientations (layer 13). In the middle-to-early layers, we see curves and lines, branching patterns, and embossments, as opposed to the earlier layers that only show shapes arranged in lines (Layer 35). In the middleto-late layers, we see more obvious objects, like faces, eyes, and architectural features. Finally, the later layers filter for more complicated higher level conceptual features that are difficult to interpret; we begin to see "part" filters [35]. We can see what appear to be facial features in the image from layer 96, geometric structures, archways, and architectural glass ceilings in the images from layer 105. We can see eye-like patterns in the image from layer 126. The repeated convolutions can make the features more difficult to optimize for, so by the 139th layer, these images become more difficult to produce. In addition, the high level conceptual features are more complex, and thus are more difficult to represent in a single image (layer 139).  Finally, we can run the feature visualization analysis on the AlexNet features ( fig. 9). All of the images are taken from the fourth out of five convolutional layers. While, in general, this technique is more difficult to perform on convolutional networks without residual connections, the images are clear enough to see what is occurring in each feature. Some filters select for orientation and edge information, and all of them show evidence of selecting for textural information.
Even with the limitations on examining AlexNet features, when comparing them to the ResNet features there is considerable evidence that AlexNet is filtering for low level perceptual features, like textures and edges and in some cases simple patterns, while the ResNet is able to filter for more conceptual objects.
There are technical limitations to analyzing the ResNet features any further. Convolutional neural networks are considered "black box models", meaning that it can be difficult to understand why a prediction is made in a particular case. While great progress in analyzing deep neural networks has been made by the community, it is still difficult to compile human-readable descriptions of the inference process. As such, it is difficult to know which features of the model and aspects of the images are associated with memorability. However, we make these computed network features publicly available on the Open Science Framework (OSF, https://osf.io/qf5ry/) to allow further investigations of these features that drive predictions of memorability.

Data Sharing
One impetus for this project is to make a network that is easily usable by psychologists, regardless of familiarity with programming or deep learning. Furthermore, the prior leading model, MemNet, is no longer available, with full code and methodological information inaccessible. Making our model easily available allows researchers across the vision and memory communities to control and select stimuli for experiments based on memorability. Further, this model may serve as a jumping point for developing more biologically inspired models of the visual memory system. We have taken several steps in order to promote the spread of ResMem.
First, ResMem is available as a Python package that can be easily installed using the command:

pip install resmem
It requires the standard Python packages Numpy [36], Torch [26], and Torchvision [33] to run, which pip will download automatically, and it requires minimal Python experience to operate. Numpy is considered a standard package for working with data of any kind, and Torch and Torchvision are the most commonly used packages for deep learning research. Second, we have created a web application at The Brain Bridge Lab Website (https://brainbridgelab.uchicago.edu/resmem/), where anyone can upload an image and quickly get back its estimated memorability score without installing any software packages. Third, the code for the network as well as the code used to create the network feature visualizations is available on GitHub at https://github.com/Brain-Bridge-Lab/resmem-analysis, and a large number of pre-calculated visualizations are available on OSF at https://osf.io/qf5ry/. This should allow researchers to view the specific architecture, weights, and biases in the network, while also allowing for explorations of how different features emerge from various network layers.

Durability
In the interest of making these resources available to the community as long as possible, we have taken measures to ensure that ResMem will be accessible even if one way to access it stops working. The model consists of two files, a python file which encodes the architecture of the model, and a PyTorch file, which contains all of the weights and biases needed to make inference using the model. One advantage of using PyTorch is that the PyTorch file is humanreadable, unlike the weights and biases for a Caffe model, which is stored in a binary file. This means that even when PyTorch is eventually outmoded, it will be easier to port the model to a new framework. In addition, while the files that define the architecture for a Caffe model are human-readable, a number of parameters are defined implicitly, so it can be difficult to fully understand the architecture without advanced deep learning experience. PyTorch largely follows the philosophy: "explicit is better than implicit", and as such PyTorch modules are easier for non-deep-learning experts to understand.
In the process of development, we used GitHub to track changes to the package, and thus the most up-to-date version of both files is stored at https://github.com/Brain-Bridge-Lab/resmem. For easy installation and distribution, ResMem is also stored on PyPI, the official Python Package Index. Normally they do not distribute pre-trained models, but we have received special dispensation to use PyPI for ResMem, and thus, the package can be easily installed with one command.
For additional resilience, we have also uploaded all of the files needed to build ResMem to an OSF repository at https://osf.io/qf5ry/. These three file distribution services (GitHub, PyPI, and OSF) are considered integral to the programming, python, and scientific communities respectively, and by distributing ResMem on all three, we expect it to be available to researchers until well after it is outmoded by future research. We have also included a step-by-step tutorial on how to start using ResMem in the Supplemental Information.

Validations of ResMem
One concern of the prior model MemNet was that the model could be overfit on the distinctive image set used (LaMem). There was also the concern that the broad range of images represented in LaMem prevented MemNet from making successful within-category predictions. In order to test the generalizability of ResMem, we successfully validated its predictions with human memory performance recorded from three other studies using entirely different image sets in different experimental paradigms.
First, we compared ResMem predictions with memory performance for black and white scene images by individuals at different stages of Alzheimer's dementia [37]. In contrast with the continuous recognition tasks used in the training sets for ResMem (LaMem and MemCat), here, participants performed a perceptual indoor/outdoor scene categorization task, and then after a 70 minute delay, had to indicate memory with old/new judgments for the images intermixed with matched foils. ResMem was significantly able to predict memory performance for all participant groups, including healthy controls (Spearman rank correlation: ρ = 0.30, p = 2.82 × 10 −19 ), those with a subjective cognitive decline (ρ = 0.25, p = 2.63 × 10 −13 ), and those with a diagnosed mild cognitive impairment, MCI (ρ = 0.14, p = 4.03 × 10 −5 ). ResMem outperforms the model reported in the original paper, which was unable to predict memory for those with MCI [37]. ResMem's success has tremendous implications for its use in making predictions about memory in patient groups; for example, ResMem could be used to design highly memorable environments for those developing dementia.
Second, we tested ResMem with a highly constrained image set, specifically the Food Folio dataset [38], a set of overhead photographs of food items on a white plate against a black background. ResMem predictions were significantly correlated with human continuous recognition task performance, ρ = 0.24, p = 0.004 [39]. This indicates that ResMem's predictions are even successful for images that are highly visually similar and within the same semantic category (food). This also shows how ResMem could be used to select items within a specific set for real-world applications-for example, choosing the most memorable food item to use in a nutritional poster.
Finally, we tested whether ResMem would replicate human memory performance scores made publicly available in prior work from another research group [40]. In the original study, Dubey and colleagues (2015) studied the relationship of the memorability of objects to scenes containing those objects.
ResMem was significantly able to predict memory performance for the scene images (ρ = 0.36, p ≈ 0). Importantly, ResMem could also predict memory for cropped images of single objects within the scenes (ρ = 0.14, p = 8.34 × 10 −14 ). ResMem thus is able to successfully replicate prior work, and could be used to test key questions about the relationships of memory across different stimulus categories (e.g., scenes and objects).
We envision more validations of ResMem will emerge over time, and have created a living resource of validations of the network (https://brainbridgelab.uchicago.edu/resmem/resmem-validation/). This will allow researchers to see which types of images they can use with ResMem to predict memorability (and which are not successfully predictable).

Deep Learning
Using modern machine learning techniques can vastly improve our ability to estimate memorability, and it also opens up new avenues of analysis. In addition to making memorability analysis simpler, models like ResMem can be leveraged to do analysis of high-level visual features, and how those features relate to memorability. As hardware technology continues to progress, more complicated models, such as graph-based neural networks, M3M without the deconvolution layers, and model-based algorithms can be employed to help understand memorability in those contexts.
In addition to showing the ability of modern techniques to estimate memorability, we have also shown that computational psychologists need not be constrained to off-the-shelf general-purpose architectures like AlexNet or even ResNet. By constructing a model specifically with memorability in mind, and taking into account known features of memorability, we have shown that when domain knowledge is incorporated into the architecture design, models will perform better than their general-purpose counterparts.

Psychology
An important goal of this research is to make ResMem available to any scientist who wants to study what makes something memorable. This model will allow researchers to easily and inexpensively control for memorability in other studies. In many cases, it is helpful to know in advance what participants will remember or forget; researchers can select stimuli that are highly powered for driving differences in memory, or stimuli that are well-controlled for memorability effects. Researchers can also analyze data post-hoc to see if the intrinsic memorability of their images accounted for any of their effects. In fact, research has shown that sensitivity to stimulus memorability during the perception of an image can account for some patterns in the brain previously attributed to memory encoding [5]. It is also possible that other work has examined stimulus category effects that could be confounded with memorability; for example, tests of face familiarity comparing celebrities versus non-celebrities could instead be examining memorable versus non-memorable faces, rather than prior visual experience. Thus, an easy method for quantifying the memorability of stimuli should be an indispensable resource for psychologists and neuroscientists across different fields.
More broadly, this model can serve as a resource for the general publicallowing educators to identify memorable materials, helping the average person choose a memorable photo to post online, and aiding clinicians in creating memorable environments for those experiencing memory loss [37]. In validations of ResMem, we found that it could significantly predict memory for patient groups experiencing the onset of dementia. Future work could use ResMem to identify the images that are most predictive of a developing deficit. We also found in our validations that ResMem can successfully predict memory for highly specific image sets, such as food images. ResMem would thus likely be able to identify which image in a set of similar images would best drive memory, for materials such as textbooks or infographics. Because ResMem is easy to use, it can be easily adapted for a wide range of these applications.
To support its easy use, ResMem is downloadable through the python package resmem, and a demonstration is available on the Brain Bridge Lab website (https://brainbridgelab.uchicago.edu/resmem).
Importantly, the features that emerge from ResMem give an initial glimpse into the properties that make an image memorable. Thus far, it has been an open mystery of what features drive the memorability of an image. A large body of prior work has shown that memorability is not synonymous with other low-level or high-level image features, such as color, edges, aesthetics, interest, or saliency [6,10,41]. In fact, there is even no correlation between what people think they will remember and what is in fact memorable [10]. The current study has begun to give a glimpse into the features that drive memorability: showing that including high-level conceptual features in the memorability estimation process improves the accuracy of those estimations considerably. In comparison with MemNet which only decomposes images into perceptual features, ResMem achieves better predictive power in terms of generalizability, rank correlation, and estimation accuracy. This, when taken in concert with previous research, implies that memorability is related intimately to both the perceptual and conceptual features of an image, and recent converging evidence in the psychological domain has also shown a dominance of semantic information in determining the memorability of an image [42]. Importantly, the fact that ResMem can successfully predict the memorability of images demonstrates that it is an image-computable feature that may be automatically processed during perception. In fact, image classification DNNs such as AlexNet show strong correspondences in representation with the human visual system [43,44], and dissimilarity in early layers and similarity in late layers has shown correspondences with scene memorability [45]. The current network may better capture higher-level visual and semantic processes; even without a "memory" built into the model, ResMem is able to predict memory performance using the processing of visual features alone. As a whole, this suggests that a visual system can be sensitive to what we are likely to remember, even in the absence of top-down processes or processes of memory encoding and retrieval. Such findings could support current hypotheses that memorability reflects how visual information is prioritized for later memory, as a type of sorting metric for perceptual inputs [9]. Future work can examine the relationship between this model and the visual and memory systems in the brain, as prior work has done for visual areas [8,43].
Carefully applying deep learning in the memorability field can help us better understand the phenomenon for years to come.

Acknowledgements
We would like to acknowledge Lore Goetschalckx for providing us with details on MemNet's implementation. We would also like to acknowledge Deepasri Prasad and Max Kramer for information sharing and general feedback.

Funding
Not applicable

Conflicts of Interest
Not applicable

Availability of data and material
The model is available from the python packaging authority (https://pypi.org/project/resmem/), and an online demo is available on the Brain Bridge Lab website ( https://brainbridgelab.uchicago.edu/resmem). Miscellaneous data, including feature analyses, prediction performance within all subcategories of MemCat, and an archival copy of the pretrained model are hosted on OSF at ( https://osf.io/qf5ry/). The data used to train ResMem came from two sources. LaMem is hosted by MIT (http://memorability.csail.mit.edu/download.html). MemCat is hosted by the Flemish government (https://gestaltrevision.be/projects/memcat/).

Package Tutorial
While the resmem package may be used in any way that the user wishes, this section will outline the recommended steps for basic usage. First, it is recommended to set up a Python(?) virtual environment. Then resmem can be installed using pip, the built-in Python package manager. Also install Pillow(?) to handle image importing. For large datasets, it is recommended to create a DataSet object using torch data utilities(?). Depending on how the dataset is organized, the following code may need to be adjusted. It is written with the assumption that there is a text file named data.csv which has image file names as the first column and their experimentally determined memorability scores as the second column, but the user may structure this however they want. For managing data, the package numpy is often used. Both Numpy and Torch are included in the resmem installation. In the following code it is aliased as np. Then, it is recommended to use the DataLoader from torch to automatically create batches. from torch . utils . data import DataLoader image_set = ResMemDataset () data = DataLoader ( image_set , batch_size =8 , num_workers =4 , pin_memory = True ) # A batch size of 8 will be a p p r o p r i a t e for most GPUs . # Pinning the memory greatly i n c r e a s e s p e r f o r m a n c e on large d a t a s e t s . ( n >> b a t c h _ s i z e ) model = ResMem ( pretrained = True ). cuda () # The . cuda () method will t r a n s f e r the model onto a GPU . # If you do not have an NVIDIA GPU , omit this method . for batch in data : x , y , name = batch ys += y . squeeze (). tolist () # Store the ground truths for later . # If your dataset does not include ground truths , omit any r e f e r e n c e s to y . bs , c , h , w = x . size () # Get all of the size and shape i n f o r m a t i o n about the data . y_hat = model ( x . cuda (). view ( -1 , c , h , w )). view ( bs , -1). mean (1) y_hats += y_hat . squeeze (). tolist () names += name rcorr = spearmanr ( ys , preds )[0] loss = (( np . array ( ys ) -np . array ( preds )) ** 2). mean ()

Model Training
We conducted our hyperparameter tuning using https://www.wandb.ai (?) ( fig. 1). Each of the lines in ( fig. 1) represents a separate training session of our reconstructed MemNet, recording the Spearman rank correlation of the model's predictions on a validation dataset after each epoch. Each attempt has different hyperparameters, or the training/testing environment was changed in some minor way. Hundreds of training and testing attempts were run, and a selection are shown in fig. 1 When examined in comparison with the validation performance of ResMem ( fig. 2), it can be seen that ResMem trains much more slowly, in most cases not even reaching an overfitting regime by the 30,000th training step. This is a feature of ResMem being a much larger, more complex model.
The hyperparameter tuning phase is done by training and testing hundreds of models with slightly tweaked hyperparameters. The hyperparameter set that is associated with the best performance in terms of Spearman rank correlation was then used as the hyperparameter set for the final training process. The model created by this process was used for all of the analyses in the main text, and in the case of ResMemRetrain, is published for use by future researchers. Fig. 1 The MemNet tuning process is shown in (a). In this figure, hyperparameters were generated by hand to explore the behavior of the model in training. One run is longer than the others to examine how the model behaves in the overfitting regime. Eta(η) represents the learning rate, and gamma(γ) represents the momentum of the optimization method. Fig. 2 Validation performance from ResMem's hyperparameter tuning phase. Each line represents a model with slightly different hyperparameters (training rates, momenta). As the training phase goes on, for the most part, performance improves on the validation set.