{'fraction_non_alphanumeric': 0.026720327241538562, 'fraction_numerical': 0.01121593982978162, 'mean_word_length': 4.834488067744418, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
DYPLODOC: Dynamic Plots for Document Classification

Narrative generation and analysis are still on the fringe of modern natural language processing yet are crucial in a variety of applications. This paper proposes a feature extraction method for plot dynamics. We present a dataset that consists of the plot descriptions for thirteen thousand TV shows alongside meta-information on their genres and dynamic plots extracted from them. We validate the proposed tool for plot dynamics extraction and discuss possible applications of this method to the tasks of narrative analysis and generation.


Introduction
Natural Language Generation (NLG) is one of the areas within Natural Language Processing (NLP). Deep learning enabled various generative applications where generated texts are short and constrained by the context. These could be dialogue responses [9,10], traditional or stylized poetry [23,26,27], end-to-end author or source stylization [6,20,22]. One of the bottle-necks that holds modern Natural Language Processing (NLP) from the generation of longer texts is the concept of narrative. There are constant attempts to generate longer blocks of text, such as [7] or [1], yet they succeed under certain stylistic and topical constraints that exclude the problem of narrative generation altogether [19].
Though philosophers and linguists try to conceptualize the notions of plot, narrative arc, action, and actor for almost a century [16,18,24], few of these theoretical concepts could be instrumental for modern NLP. Ostermann et al. [13] present a machine comprehension corpus for the end-to-end evaluation of script knowledge. The authors demonstrate that though the task is not challenging to humans, existing machine comprehension models fail to perform well, even if they make use of a commonsense knowledge base. Despite these discouraging results, there are various attempts to advance narrative generation within the NLP community, see [2,3]. For a detailed review of approaches to narrative generation, we address the reader to [8], here we just mention several ideas that are relevant within the scope of this paper.
One line of thought is centered around some form of a multi-agent system with various constraints that keep generated texts within some proximity of human-written narrative. For example, Theune et al. [21] present a multi-agent framework for automatic story generation where plots are automatically created by semi-autonomous characters while the resulting plot is converted into text by a narrator agent. Walker [25] experiments with a dialogic form of a story: a deep representation of a story is converted into a dialog that unfolds between two agents with distinctive stylistic characteristics. Such multiagent-based systems show a lot of potential yet do not leverage the existing datasets that include an extensive amount of narratives written by humans.
Another line of research tries to use some NLP methods and statistical procedures to systematize existing narratives using some form of synthetic features that could in some form describe the narrative structure of a given story. Pérez y Pérez [15] represents a story as a cluster of emotional links and tensions between characters that progress over storytime. Schmidt [17] introduces a concept of narrative arc in a context of large-scale text mining. The author conceptualizes plot structure as a path through a multidimensional space derived from a topic model. This approach allows us to see plot dynamics within any given book or show in detail, yet it is harder to generalize. Topic modeling tends to provide profoundly different sets of topics across genres. When aggregating across all topics within a dataset one gets a high dimensional space that is harder to interpret. For example, similar narrative arcs of a detective story would look different in terms of topic modeling if the same plot takes place in Victorian England or on Mars. One would like to keep the concept of a narrative arc yet to make it more abstract in order to be able to carry comparative narrative research across genres and settings. Mathewson et al. [12] develop ideas that address this challenge to a certain extent. The authors mathematize a concept of narrative arc that depicts the evolution of an agent's belief over a set of the so-called universes. The authors estimate these probabilities with pre-trained probabilistic topic classifiers and combine these estimates with a multi-agent dialogic approach. They demonstrate that the agent that tracks the dynamics of such narrative arcs generates more engaging dialog responses. This paper shows that the concept of the narrative arc could be applied to a vast variety of narratives in natural languages and does not have to be limited to the setting of a dialogue system. The contribution of this paper is two-fold: (1) we provide a large dataset of longer storylines that could be used for narrative research and (2) use it to demonstrate how the notion of the narrative arc could be applied to longer texts and provide meaningful insights into the structure of the story. This structural insight could be useful both for quantitative research of narrative dynamics and as synthetic features for the generation of narrative structures in the future.

Data
TVmaze 1 is a huge database with the data on different TV shows that include the storyline as well as certain meta information. Using TVmaze API we have obtained 300+ Mb of data that could be found online as a part of the DYPLODOC package 2 . The dataset includes 13 813 TV shows and 21 962 seasons that consist of more than 300 000 episodes. Every episode in the dataset is annotated with a condensed description of the main storyline and the narrative structure of a season is defined as a sequence of such descriptions. The data includes main characters, that could be extracted with the NER algorithm, the majority of texts are centered around most reportable events [14], and the narrative is split into clear steps, certain 'quanta' of a story. Almost every season in TVmaze has a tag of a genre, such as (Horror, Food, Drama, etc). We hope that the publication of the dataset could facilitate further quantitative research of narrative structures.

Plot Dynamics Extraction Tool
Here we leverage the fact that episode descriptions in the TVmaze dataset form seasons and demonstrate how one could build the narrative arc of a season by constructing artificial features that would to a certain degree describe the season's plot dynamics. We call this method Plot Dynamics Extraction (PDE) Tool.
Let us regard a specific TV show as a set of seasons : Every season could be represented as a sequence of episodes : Every is a description of one particular episode.
We investigated the plot dynamics using two different pre-trained probabilistic topic classifiers. However, unlike [17] we choose fixed topics that form a universal state space of narrative arcs. One could discuss if the provided list of dimensions is complete, yet we believe that two classifiers are sufficient to extract non-trivial narrative arcs out of a variety of texts: • Classifier of movie genres trained on Wikipedia film synopses provided in [5]. The labels of genres include: Comedy, Drama, Western, Adventure, Animation, Action, Thriller, Family, Romance, Fantasy, Horror, History, Music, Sci-Fi, War, Crime, Musical, Biography, Mystery, Sport. • DeepMoji: Deep neural network that was trained on a corpus of text that included 64 emojis, see [4]. In their blogpost 3 the authors suggest hierarchic aggregation of 64 emojis. Using this aggregation we limit ourselves to 11 labels in this work. These labels include: Love, Happy, Wink, Deal, Force, Eyes, Fear, Mad, Sad, Music, Misc. Thus every narrative arc is a trajectory on a union of two simplexes: a 20-simplex of genres and an 11-simplex of emotions. For the vast majority of the cases, the classifier mostly detects one or two most dominant genres and topics respectively, limiting the meaningful development of a given narrative art to one plane on each simplex.
Similarly to [12] we propose to regard a finite set of universes, U. One could develop a universe model to assess the compatibility of an episode with a given universe, ∈ U. Given such a model, one could update the posterior universe distribution over a sequence of episodes . For each universe , the universe model assigns a likelihood ( | 1: −1 , ) to an episode , conditioned on previous episodes 1: −1 of the season. Introducing ( ) as the prior distribution over universes, the conditional probability ( | ) could be defined as follows ( | ) = ( ) × ( | ) ( ) . Using this notation [12] obtain the following approximation In order to guarantee that the sequence of ( | 1: ) forms a series of coordinates on the union of 20-simplex of genres and 11-simplex of emoji, we introduce two additional smoothing procedures. First, we control ( ) distribution update so that it does not degenerate into zero, since in this case Equation (1) becomes obsolete. We also use exponential smoothing average instead of standard multiplicative update. Figure 1 illustrates the resulting dynamics of one season of 'BoJack Horseman', extracted with a Deepmoji classifier and with a classifier pre-trained on the genres 4 . There are four major non-zero components. On the emoji-simplex the narrative is balancing between MAD and SAD, though the former is prevailing throughout the season, the latter peaks in the episode when BoJack's daughter overdoses and is found in the hospital. On the genre-simplex, the narrative is balancing between Comedy and Drama. BoJack Horsemen is a dark comedy animation series so it stands to reason that Comedy dominates the season while Drama peaks in the second quarter of the season when BoJack spends time with his elderly mother in the nursing home.

Experimenting with PDE Tool
Though scores of pre-trained deepmoji and genre classifiers have very intuitive interpretation it is hard to claim that they could be useful for any form of comparative narrative research. The dynamic aspect of these constructed features is clear yet it is not obvious that their dynamics are meaningful in any non-trivial sense. We compare dynamic plots extracted from the season description with static meta-information that is included in the TVmaze data 5 . If different episodes of the season are labeled with a different tag we choose the most frequent tag as the tag for the whole season. If the show was tagged with two values 6 , we treated it like one static tag. For example, 'BoJack Horseman' has Drama | Comedy as a genre, thus Drama, Comedy and Drama | Comedy for three separate static tags.
PDE-tool represents every plot as an array of tags (such as genres or emoticons), in which every tag is represented as a sequence of probability estimations ranging from 0 to 1. We concatenate these representations with extra synthetic features, namely: mean, median, minimum, maximum, and variance for the probability of every tag; area under the curve; the moment of intersection with another tag probability estimation, etc. Thus we represent every text as a multidimensional vector. Using these dynamic based features we clustered the seasons, applying standard K-means clustering over a PCA 7 . We have carried our experiments 5 7911 seasons out of 21962 have no tags so we exclude them from this experiment. 6 We filtered the shows that had more than 2 tags for their genre. 7 Since the length of the seasons might differ and we are interested in comparative research of the obtained arcs we scale every series to the length across all possible combinations of features, a different number of principal components (50, 100, 200, 300), and a various number of clusters (50, 75, 100, 150, 200, 250, 300). Then we calculated mutual information between obtained clusters and static tags for the corresponding seasons. The higher values of mutual information were obtained for the following setup: synthetic features included statistics and area under the curve for every component; the number of principal components equals 200; the number of clusters for K-means equals 200.
With this setup, the mutual information between clusters based on both classifier combined and static tags was 0.57, as shown in Table 1. One could see that both the dynamics based on the genre classifier and the emoji classifier-based dynamics allow inferring static genre tags. The combination of two classifiers yields higher mutual information. Dynamic plots extracted by different classifiers differ significantly and go beyond static genre tags.  Figure 2 shows the resulting t-SNE (t-distributed stochastic neighbor embedding) visualization, see [11] for details, of the clustering based on plot dynamics. The centers of the clusters are labeled with static genre tags. The average size of ten episodes either uniformly stretching or compressing the resulting timeline. In some clusters the share of the dominant tag reaches 100%, the lowest possible percentage of the dominant tag is 42%. Table 3 describes basic parameters of the cluster distribution.

Discussion
Using synthetic features based on the PDE tool one could partially infer static human labels for season's genre.  Table 3. Minimum, median, average and maximum values for the size and percentage of the dominant tags in the resulting clusterization based on the dynamic plots.
descriptions of the seasons. Table 2 shows the three biggest resulting clusters with the highest percentage of one static genre tags.
The first cluster could be attributed to darker comedies. Figure 1 shown above is a good representative of this cluster. The second cluster could be attributed to historic dramas. The cluster is characterized by the dominance of Drama and interplay of Sad and Mad emojis. Figure 4 illustrates it with the dynamics of the second season of Victoria. The third cluster is also dominated by Drama but with the certain presence of Comedy as well as with domination of Mad over Sad, see Figure 5 for an example. These are only a few examples that illustrate how rich is the information provided by the PDE tool and its potential for quantitative analysis of narrative.

Conclusion
This work makes the following contributions: (1) a universal feature extraction method for plot dynamics; (2) a dataset of natural language plot descriptions; (3) finally, the paper shows that plot dynamics features extracted with the proposed method are meaningful and can shed light on the general structure of the narrative.
We hope that this paper could help to revive the attention of the community to the problems of narrative analysis and generation at the very least by providing data and tools for further exploration. We also hope to use narrative arch extracted with PDE tool as a form of semi-supervision for narrative-oriented generative models. The extracted dynamic plots are presented alongside the textual data. We hope that the proposed tool would inspire further research on plot dynamics and inform the discussions on narrative structure, particularly in the context of narrative generation.