{'fraction_non_alphanumeric': 0.02763744427934621, 'fraction_numerical': 0.012332838038632986, 'mean_word_length': 4.968371268105232, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Analysing the Memorability of a Procedural Crime-Drama TV Series, CSI

We investigate the memorability of a 5-season span of a popular crime-drama TV series, CSI, through the application of a vision transformer fine-tuned on the task of predicting video memorability. By investigating the popular genre of crime-drama TV through the use of a detailed annotated corpus combined with video memorability scores, we show how to extrapolate meaning from the memorability scores generated on video shots. We perform a quantitative analysis to relate video shot memorability to a variety of aspects of the show. The insights we present in this paper illustrate the importance of video memorability in applications which use multimedia in areas like education, marketing, indexing, as well as in the case here namely TV and film production.


INTRODUCTION
Memorability is a subconscious measure of importance imposed on external stimuli by our internal cognition which ultimately decides the experiences we either carry with us into our long-term memory, or we discard. Memory is an unequivocally unreliable phenomenon for any person and despite its importance, it is challenging for us to influence what we will ultimately remember or forget. This absence of any unified meta-cognitive insight brings meaning to the study of memory, or more specifically into the study of memorability, more generally known as the likelihood that something will be remembered or forgotten [46]. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s). CBMI 2022, September [14][15][16]2022, Graz, Austria © 2022 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-9720-9/22/09. https://doi.org /10.1145/3549555.3549592 We have evolved to store the important moments in life while forgetting uninteresting or redundant information. Thus, a memorabilitybased model of our experiences could allow us to filter and sort information we encounter using a human-memorability criterion [29]. Researching how our memories take shape could have a profound impact on a plethora of research areas such as human health or the curation of educational content [12,24].
The study of memorability may also have a commercial impact in fields such as marketing, film or TV production, or content retrieval [49]. The latter of these use-cases serves as a particularly important application of memorability as the volume of digital media content available grows exponentially over time via platforms such as social networks, search engines, and recommender systems [29,40,43].
In this research, crime-drama exemplified in television (TV) programs such as CSI: Crime Scene Investigation (referred to as CSI) acts as an experimental test bed in which computational memorability as a surrogate for cognitive processing, can be analysed. CSI is a procedural crime-drama TV series with each episode following the same format. Viewers are introduced to a crime at the beginning of an episode. As the episode progresses, various clues are revealed, concluding by revealing the true perpetrator of the crime. The success of CSI can be attributed partly to its aesthetic high production values which many media critics comment on, and we argue its success is also due to its ability to promote critical thinking and memory in its viewers.
Previous studies have already proven crime-drama TV series to be useful as test cases in multiple research areas including NLP, summarisation and multi-modal inference [17,30,31]. In particular, the work in [37] is a large-scale anthology of Barry Salt's essays on statistical analysis of film over many years of investigation. Salt has single-handedly established statistical style analysis as a research paradigm in film studies. He argues that "many directors have sharply different styles that are easily recognized" [37, p. 13]. Salt's more recent work includes the application of film theory to the creation of TV series and in [36] he presents a stylistic analysis of episodes in twenty TV drama series, finding that the structure of TV style is uniform, a similar finding to movies.
The work by Salt and others uses a variety of measures such as shot duration and shot distribution and since then several further studies have attempted to perform statistical analysis of TV content [3,7,23,35,38]. Yet none of this, or any other, forms of analysis considers the memorability of the visual content in its analysis.
We investigate the distribution of computational memorability scores of CSI episodes over 5 seasons by leveraging modern vision transformer architectures [4,13,34] fine-tuned on the task of media memorability prediction. We interleave our predicted memorability scores for given episodes of CSI with a detailed annotated corpus to investigate correlation between memorability and aspects of the show including scene and character significance.
Our research investigates whether there is a relationship between the memorability of a shot/scene and the characters present in it, how the memorability associated with particular characters develops over multiple episodes and seasons, and whether the memorability of a shot/scene is correlated with the significance of the scene. The significance of this and the reason we study the CSI TV series is precisely because of its importance in popular culture. Since its first broadcast in the early 2000s it has led to the notion of the "CSI Effect" which has altered public understanding of forensic science [27,39] and thus has had a societal impact. CSI has also been the subject of much investigation in media studies because of its importance.
The remainder of the paper is structured as follows: In Section 2, we discuss some of the seminal related work and in Section 3, we discuss the CSI dataset used throughout this study and describe the data manipulation and augmentation techniques we performed. Section 4 describes our experiments and discusses the results of these experiments. Finally, in Section 5, we make some closing remarks and discuss future directions.
It is important for the reader to note that this work does not include any direct comparison to the work of others because there is nothing for us to compare against. Thus a direct evaluation of this work is not possible because of the nature of the topic. Instead, our novel findings present the kinds of insights which can be gleaned from computational memorability analysis of visual content and this points to the promise that such analysis can have in areas like educational content, marketing, film and TV production, and content retrieval.

RELATED WORK 2.1 Media Memorability
Video Memorability (VM) is a natural progression from a related research discipline: image memorability (IM). The work of [21] is regarded as a seminal paper on IM in which it is stated that memorability is an intrinsic property of an image, an abstract concept similar to image aesthetics which can be computed automatically using modern image analysis techniques [20]. The use of deeplearning frameworks has led to results at near human levels of consistency in the task of IM prediction [2,6,15] partly attributed to the development of image datasets with annotated memorability scores [22]. In contrast, VM is still in its early stages and only recently has there been available data sets [8,29] on which to train and test VM models.
The Mediaeval Predicting Media Memorability task first took place in 2018, has run annually since then [9,11,12,24,25] and has been central to the development of VM. The 2021 edition saw the use of vision transformers [4,13,34] across all participant submissions. The focus of work in the area is predominantly on developing memorability prediction techniques and to date the most applicable use of VM has been in the analysis of TV advertisements [28,41].

Affective Video Content Analysis
Affective Video Content Analysis (AVCA) is a research area closely related to VM which investigates the emotions elicited by video among its viewers [5]. Like VM, AVCA-based research aims to generate metacognitive insights capable of improving video content indexing and retrieval [19,50]. Besides this, AVCA research could, for example, protect children from emotionally harmful media [48], as well as create mood-dependent video recommendation [18]. Past research in this domain has shown the modelling of some signatures of film such as tempo [1], violence [14,32], and emotion [45].

Macabre Fascination in CSI: The Rise of the Corpse
The impact of crime-drama TV series such as CSI on modern popculture stretches far beyond academic research, even resulting in the coining of a new term: 'The CSI Effect' [27,39], mentioned earlier. Forensic autopsy portrayed in CSI acts as a softening lens in which exploration of the dead who are usually the victims of the crime which is the focus of the episode, is socially-palatable entertainment [33]. As CSI viewership increased from the first series in 2001, so did the number of forensic science courses available across educational bodies 1 . Early 2000s crime-drama TV such as CSI sexualises the cadavers present in their series through the everpresent "beautiful female victim" motif. The sexualised corpse is a stock character that makes regular appearances. Our eyes are invited to linger on the passive, beautiful, dead bodies that lay in the morgue. This phenomenon captured the public's attention and increased the abundance of sexualised cadavers throughout subsequent seasons of CSI while other series such as Law & Order began adding autopsies to their shows [16].

THE CSI DATASET 3.1 Source Data
We use 39 annotated episodes of CSI in our analysis, curated by Frermann et al. [17] and available on the Edinburgh NLP GitHub 2 . The original source data consists of two corpora, the first of which is Perpetrator Identification (PI). PI corpus' files are annotated at word-level for each episode. A word can be uttered by a speaking character or is part of the screenplay. Each word is accompanied by case ID, sentence ID, speaker, sentence start time, sentence end time.
We also have binary indicators such as killer gold, suspect gold, and other gold which indicate whether the word uttered mentions the killer, suspect, or otherwise. The second corpus in the original annotated data set is Screenplay Summarisation (SS) presented at scene-level (a section of the story with its own unique combination of setting, character, and dialogue). For a given episode, the dataset consists of each scene annotated with Scene ID, screenplay, as well as the scene aspect. Aspect is a categorical feature indicating the scene's significance and can be one or many of the following: Crime Scene, Victim, Death Cause, Evidence, Perpetrator, Motive, or None.
Each of the 39 episodes was also available to us in video format by scraping directly from the DVDs purchased from Amazon, as well as an extra 3 episode video files.

Data Manipulation:
We aggregated each of our PI episode datasets from word-level to sentence-level, combining the killer gold, suspect gold, and other gold binary indicators previously mentioned into a single categorical column. We then disaggregated our SS corpora from scene-level to sentence-level. With equivalent datasets of the same configuration, we then merged them and Table 1 shows a section of one of our augmented and merged datasets for S1.E8.

Data Augmentation:
Features available from the annotation are rich and allow us to analyse data in a variety of ways. However, we noted two shortcomings: (i) the data provides little context regarding suspects throughout an episode other than the type mentioned feature and (ii) the aspect feature does not allow us to identify scenes involving suspects. Also, a portion of scenes involving the killer are not labelled with the 'Perpetrator' aspect. We extended the aspect feature of any sentence in a scene in which the perpetrator or a suspect speaks so as to have the values of 'Perpetrator' or 'Suspect', respectively.

Pre-processing of Video Files
We subdivided our clips into shots which allows for more specific and fine-grained indexing, in turn reducing the amount of video we needed to process per analysis. We used a neural shot-boundarydetection (SBD) framework, TransNet V2, 3 described in [44], to segment episodes into individual shots.

Data Generation
The use of vision transformers in predicting memorability scores is shown in [10,26]. Our vision transformer architecture is the CLIP [34] pre-trained image encoder. We use this encoder to train a Bayesian Ridge Regressor (BRR) on the Memento10k dataset [29]. BRR was used previously by [46] for computing memorability scores based on extracted features. For each training sample, the encoder extracts features from frames sampled at a rate of 3 frames per second. The BRR is then trained on the features and memorability score pairs. To compute the memorability of unseen CSI clips, we first extract representative frames using a temporalbased frame extraction mechanism. These are used as input to the vision transformer architecture to extract features. Our BRR model then produces a memorability score based on these features. This framework is shown in Figure 1.

METHODOLOGY AND EXPERIMENTAL RESULTS
We now present the distribution of memorability scores across each episode, relating it to its metadata from annotations taken from [17].
For each episode, we plotted the memorability scores associated with shots in chronological order to create a time-series signal representing the intrinsic memorability of the episode. We then annotated each episode's signal according to various metadata including (i) the scene aspect, and (ii) the character speaking (including None). We used Laplacian smoothing described in Equation 1 to remove the jaggedness present in raw memorability scoreŝ where is the size of the smoothing window. We generated signals for each episode using a range of smoothing window sizes from 15 to 305 shots.
To compliment the shot memorability score distributions we gathered memorability scores across the distribution of characters, and of aspects. By aggregating the overall speaking time of the main characters, we identify the lead as well as the secondary characters. We performed a similar analysis to understand the scene count per aspect value. Our results are shown in Figure 2.
We investigated the memorability distribution associated with the series characters and aspects and this is presented in Figure 3 parts (a) and (b) in each individual season as well as across all 5 seasons. We investigated: i)the main character speaking, and (ii) the scene aspect. The lack of variation across Figure 3 (a) and (b) and the relatively high memorability scores is an artefact of the Memento10k memorability dataset [29] used to train the memorability model we used where we rarely see a video segment given a memorability score below 0.7. Instead we are interested in repeating patterns e.g. which parts of shows consistently result in higher or lower memorability scores and what these patterns might reveal to us about the series.
In Figure 3 (a), we see that ordering the cast in terms of memorability over all 5 seasons is almost identical to that seen in Figure 2 (a). Catherine has less speaking time than Grissom, but is consistently considered more memorable. With the exception of Season 3, Nick is simultaneously the least-memorable character and the character with the least time spent speaking. We observe a somewhat positive linear relationship between a character's importance (in terms of screen-time), and their memorability.
In Figure 2 (b), the proportion of scenes in which each aspect value appears correlates with the purpose in which that aspect serves. For example, 'Motive' appears least across all seasons as only a small portion near the end of each episode is dedicated to revealing a perpetrator's motive. Similarly, 'Crime scene' and 'Victim' scenes usually occur near the beginning of an episode. Despite this, as shown in Figure 3 (b), 'Motive' scenes are considered the most memorable across all 5 seasons. In contrast, both 'Crime scene' and 'Victim' scenes are considered the least memorable.
Scenes described with the 'Death Cause' aspect value are associated with autopsy scenes in which viewers are shown flashbacks to the crime committed, as well as the cadaver of a usually youthful, attractive victim. We discussed the importance of these scenes and their pivotal role in the development of TV crime series in Section 2.3. In Figure 3 (b), the intrinsic memorability of this scene type increases as the seasons progress. These pedagogic, grotesque, scenes capture the audience's fascination because of their brash   autoptic vision [47] and are a staple of the crime-drama TV genre and contribute to its popular success. The visual memorability of these scenes correlates with the success of this scene-type.

CONCLUSIONS AND FUTURE WORK
Video memorability is a recent computational tool for analysis of visual media which is more abstract and higher-level than raw content analysis like object detection or action classification. In this paper we have used the computation of video memorability at video shot level to analyse a popular procedural crime-drama TV series revealing insights not previously visible.
Our investigation creates a mélange of intertwined arguments ranging from more statistically-driven interpretations of film such as cinemetrics, to artistic studies on the relationship between a series' popularity and its most memorable moments. We hypothesise that scene and character importance is related to the memorability of a scene, and show that correlation exists between factors of the show not previously visible. We show that these correlations are not random, but instead have complex interpretation. The memorability scores we generated uncovered patterns within the early seasons of CSI, highlighting the significance and memorability of an episode's finale in comparison to other scenes. Our memorability  scores for lead cast members heavily correlate with the characters' importance, considering on-screen time as importance. The work here is one of the first efforts in the application of memorability scores beyond benchmarking environments.
Our first area for future work is to increase the data size by including more episodes of CSI, as well as episodes from other shows such as CSI: Miami or other 'Whodunnit-styled' series such as Law & Order. In our study, we used a temporal-based frame extraction method to extract representative frames from shots. This could be enhanced via Representative Frame Extraction techniques [42], which are capable of automatically selecting the most representative frames from video content.
It is clear that we have developed an importance-proxy by analysing various metadata from CSI. Ideally, we would define a user task in which participants' genuine memorability of aspects of a crimedrama series can be examined. This gives a real measure of importance with respect to the show, rather than an importance proxy.
The novel findings we have presented here reveal insights from computational memorability scores for video shots of a TV crime series. As mentioned earlier, this is useful and interesting from a film or TV studies perspective but it also shows the potential that computational memorability has in other areas. We could now analyse or even edit videos to be used in online education so as to maximise the memorability of key moments. We could structure video content used in marketing and advertising, or as we have seen here in film and TV production, so that key messages are delivered with maximised memorability. Future work in these and other areas may indeed exploit computational memorability as a criterion for composing video content.