{'fraction_non_alphanumeric': 0.0447906976744186, 'fraction_numerical': 0.011162790697674419, 'mean_word_length': 4.522989982019008, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Bayes Factor Scoring for Authorship Verification

The PAN 2020 authorship verification (AV) challenge focuses on a cross-topic/closed-set AV task over a collection of fanfiction texts. Fanfiction is a fan-written extension of a storyline in which a so-called fandom topic describes the principal subject of the document. The data provided in the PAN 2020 AV task is quite challenging because authors of texts across multiple/different fandom topics are included. In this work, we present a hierarchical fusion of two well-known approaches into a single end-to-end learning procedure: A deep metric learning framework at the bottom aims to learn a pseudo-metric that maps a document of variable length onto a fixed-sized feature vector. At the top, we incorporate a probabilistic layer to perform Bayes factor scoring in the learned metric space. We also provide text preprocessing strategies to deal with the cross-topic issue.


Introduction
The task of (pairwise) authorship verification (AV) is to decide if two texts were written by the same person or not. AV is traditionally performed by linguists who aim to uncover the authorship of anonymously written texts by inferring author-specific characteristics from the texts [11]. Such characteristics are represented by so-called linguistic features. They are derived from an analysis of errors (e.g. spelling mistakes), textual idiosyncrasies (e.g. grammatical inconsistencies) and stylistic patterns [11].
Automated (machine-learning-based) systems have traditionally relied on so-called stylometric features [20]. Stylometric features tend to rely largely on linguistically motivated/inspired metrics. The disadvantage of stylometric features is that their reliability is typically diminished when applied to texts with large topical variations.
Deep learning systems, on the other hand, can be developed to automatically learn neural features in an end-to-end manner [5]. While these features can be learned in such a way that they are largely insensitive to the topic, on the negative side, they are generally not linguistically interpretable.
In this work we propose a substantial extension of our published ADHOMINEM approach [4], in which we interpret the neural features produced by ADHOMINEM not just from a metric point of view but, additionally, from a probabilistic point of view.
With our modification of ADHOMINEM we were also cognizant of the proposed future AV shared tasks of the PAN organization [16]. Three broader research questions (cross-topic verification, open-set verification, and "surprise task") are put into the spotlight over the next three years. In light of these challenges we define requirements for automatically extracted neural features as follows: -Distinctiveness: Our extracted neural features should contain all necessary information w.r.t. the writing style, such that a verification system is able to distinguish same/different author/s in an open-set scenario. In order to automatically quantify deviations from the standard language, the text sample collection for the training phase must be sufficiently long. -Invariance: Authors tend to shift the characteristics of their writing according to their situational disposition (e.g. their emotional state) and the topic of the text/discourse. Extracted neural features should therefore, ideally, be invariant w.r.t. the topic, the sentiment, the emotional state of the writer, and so forth. -Robustness: The writing style of a text can be influenced, for example, by a desire to imitate another author (e.g. the original author of a fandom topic) or by applying a deliberate obfuscation strategy for other reasons. Our extracted neural features should still lead to reliable verification results, even when obfuscation/imitation strategies are applied by the author. -Adaptability: The writing style is generally also affected by the type of the text, which is called genre. People change their linguistic register depending on the genre that they write in. This, in turn, leads to significant changes in the characteristics of the resulting text. For a technical system, it is thus extremely difficult to establish a common authorship between a WhatsApp message and a formal job application for example. In forensic disciplines, it is therefore important to train classifiers only on one genre at a time. In research, however, it is quite an interesting question how to, e.g., find a joint subspace representation/embedding for text samples across different genres.
We assume that a single text sample has been written by a single person. If necessary, we need to examine a collaborative authorship in advance [11]. Dealing with genreadaption or obfuscation/imitation strategies is not part of the PAN 2020/21 AV task. Another open question is the minimum size of a text sample required to obtain reliable output predictions. This question will also be left for future work.

ADHOMINEM: Siamese network for representation learning
Existing AV algorithms can be taxonomically grouped w.r.t. their design and characteristics, e.g. instance-vs. profile-based paradigms, intrinsic vs. extrinsic methods [18], or unary vs. binary classification [12]. We may roughly describe the work flow for a traditional binary AV classifier design as follows: In the feature engineering process, a set of manually defined stylometric features is extracted. Afterwards, a training and/or development set is used to fit a model to the data and to tune possible hyper-parameters of the model. Typically, an additional calibration step is necessary to transform scores provided by the model into appropriate probability estimates. Our modified ADHOMINEM system works differently. We define a deep-learning model architecture with all of its hyper-parameters and thresholds a-priori and let the model learn suitable features for the provided setup on its own. As with most deep-learning approaches, the success of the proposed setup depends heavily on the availability of a large collection of text samples with many examples of representative variations in writing style.
The majority of published papers, using deep neural networks to build an AV framework, have employed a classification loss [1], [17]. However, metric learning objectives present a promising alternative [5], [8]. The discriminative power of our proposed AV method stems from a fusion of two well-known approaches into a single joint end-toend learning procedure: A precursor of our ADHOMINEM system [4] is used as a deep metric learning framework [14] to measure the similarity between two text samples. The features that are implicitly produced by the ADHOMINEM system are then fed into a probabilistic linear discriminant analysis (PLDA) layer [9] that functions as a pairwise discriminator to perform Bayes factor scoring in the learned metric space.

Neural extraction of linguistic embedding vectors
A text sample can be understood as a hierarchical structure of ordered discrete elements: It consists of a list of ordered sentences. Each sentence consists of an ordered list of tokens. Again, each token consists of an ordered list of characters. The purpose of ADHOMINEM is to map a document to a feature vector. More specifically, its Siamese topology includes a hierarchical neural feature extraction, which encodes the stylistic characteristics of a pair of documents (D 1 , D 2 ), each of variable length, into a pair of fixed-length linguistic embedding vectors (LEVs) y i : where D denotes the dimension of the LEVs and θ contains all trainable parameters. It is called a Siamese network because both documents D 1 and D 2 are mapped through the exact same function A θ (·). The internal structure of A θ (·) is illustrated in Fig. 1. After preprocessing and tokenization (which will be explained in Section 3), the system passes a fusion of token and character embeddings into a two-tiered bidirectional LSTM [13] network with attentions [2]. We incorporate a characters-to-word encoding layer to take the specific uses of prefixes and suffixes as well as spelling errors into account. An incorporation of attention layers allows us to visualize words and sentences that have been marked as "highly significant" by the system. As shown in Fig. 1, the network produces document embeddings, which are converted into LEVs via a fullyconnected dense layer. With this output layer, we can control the output dimension. AV is accomplished by computing the Euclidean distance [14] d(D 1 , between both LEVs. If the distance in Eq. (2) is above a given threshold τ , then the system decides on different-authors, if the distance is below τ , then the system decides on same-authors. Details are comprehensively described in [4].
Pseudo-metric: ADHOMINEM provides a framework to learn a pseuo-metric. Since we are using the Euclidean distance in Eq. (2) we have the following properties: Note that we may obtain d(D 1 ,  Figure 1. Flowchart of the neural feature extraction as described in [4]. A given text sample is transformed into the learned metric space by the function A θ (·).

Loss function:
The entire network is trained end-to-end. For the pseudo-metric learning objective, we choose the modified contrastive loss [5]: where l ∈ {0, 1}, τ s < τ d and τ = 1 2 (τ s + τ d ). During training, all distances between same-author pairs are forced to stay below the lower of the two thresholds, τ s . Conversely, distances between different-authors pairs are forced to remain above the higher threshold τ d . By employing this dual threshold strategy, the system is made more insensitive to topical or intra-author variations between documents [14], [4].

Two-covariance model for Bayes factor scoring
Text samples are characterized by a high variability. Statistical hypothesis tests can help to quantify the outputs/scores of our algorithm and to decide whether to accept or reject the decision. ADHOMINEM can be extended with a framework for statistical hypothesis testing. More precisely, we are interested in the AV problem where, given the LEVs of two documents, we have to decide for one of two hypotheses: H s : The two documents were written by the same person, The two documents were written by two different persons.
In the following, we will describe a particular case of the well-known probabilistic linear discriminant analysis (PLDA) [15], which is also known as the two-covariance model [9]. Let us assume, the author's writing style is represented by a vector x. We suppose that our (noisy) observed LEV y = A θ (D) stems from a Gaussian generative model that can be decomposed as where characterizes residual noise, caused by thematic varitions or by significant changes in the process of text production for instance. The idea behind this factor analysis is that the writing characteristics of the author, measured in the observed LEV y, lie in a latent variable x. The probability density functions for x and in Eq. (4) are defined as in [6]: where B −1 defines the between-author covariance matrix and W −1 denotes the withinauthor covariance matrix. As mentioned in [7], the idea is to model inter-author variability (with the covariance matrix B −1 ) and intra-author variability (with the covariance matrix W −1 ). From Eqs. (5)−(6), it can be deduced that the conditional density function is given by [6]: Assuming we have a set of n LEVs, Y = {y 1 , . . . y n }, verifiably associated to the same author, then we can compute the posterior (see Theorem 1 on page 175 in [10]): where L = B + nW and γ = Bµ + W n i=1 y i . Let us now consider the process of generating two linguistic embedding vector (LEV) y i , i ∈ {1, 2}. We have to distinguish between same-author and different-author pairs: Same-author pair: In the case of a same-author pair, a single latent vector x 0 representing the author's writing style is generated from the prior p(x) in Eq. (5) and both LEVs y i , i ∈ {1, 2} are generated from p(y|x 0 ) in Eq. (7). The joint probability density function is then given by The term p(x 0 |y 1 , y 2 ) can be computed using Eq. (8).
Different-authors pair: For a different-authors pair, two latent vectors, x i for i ∈ {1, 2}, representing two different authors' writing characteristics, are independently generated from p(x) in Eq. (5). The corresponding LEVs y i are generated from p(y|x i ) in Eq. (7). The joint probability density function is then given by The terms p(x 1 |y 1 ) and p(x 2 |y 2 ) are again obtained from Eq. (8).
Verification score: The described probabilistic model involves two steps: a training phase to learn the parameters of the Gaussian distributions in Eqs. (5)−(6) and a verification phase to infer whether both text samples come from the same author.
For both steps, we need to define the verification score, which can now be calculated as the log-likelihood ratio between the two hypotheses H s and H d : Eq. (11) is often called the Bayes factor. Since p(y 1 , y 2 |H s ) in Eq. (9) and p(y 1 , y 2 |H d ) in Eq. (10) are independent of x 0 and x 1 , x 2 , we can choose any values for the latent variables, as long as the denominator is non-zero [6]. Substituting Eqs. (5), (6), (7), (8) in Eq. (11) and selecting x 0 = x 1 = x 2 = 0, we obtain [9] score(y 1 , where L 1,2 = B+2W , γ 1,2 = Bµ+W (y 1 +y 2 ) and L i = B+W , γ i = Bµ+W y i for i ∈ {1, 2}. As described in [6], the score in Eq. (12) can now be rewritten as where the parameters Λ, Γ , ρ and κ of the quadratic function in Eq. (13) are given by and the auxiliary variables are Hence, the verification score is a symmetric quadratic function of the LEVs. The probability for a same-author trial can be computed from the log-likelihood ratio score as follows: The AV datasets provided by the PAN organizers are balanced w.r.t. authorship labels. Hence, we can assume p(H s ) = p(H d ) = 1 2 . We can rewrite Eq. (14) as follows: Loss function: To learn the probabilistic layer, we incorporate Eq. (15) into the binary cross entropy: where φ = W , B, µ contains the trainable parameters of the probabilistic layer.
Cholesky decomposition for numerically stable covariance training: We can treat φ = W , B, µ given by Eqs.

Ensemble inference
Neural networks are randomly initialized, trained on the same, but shuffled data and affected by regularization techniques like dropout. Hence, they will find a different set of weights/biases each time, which in turn produces different predictions. To reduce the variance, we propose to train an ensemble of models and to combine the predictions of Eq. (15) from these models, where M i indicates the i-th trained model. Finally, we determine the non-answers for predicted probabilities, i.e. E p(H s |y 1 , y 2 ) = 0.5, if 0.5 − δ < E p(H s |y 1 , y 2 ) < 0.5 + δ. Parameter δ can be found by applying a simple grid search.

Text preprocessing strategies
The 2020 edition of the PAN authorship verification task focuses on fanfiction texts, fictional texts written by fans of previous, original literary works that have become popular like "Harry Potter". Usually, authors of fanfiction preserve core elements of the storyline by reusing main characters and settings. Nevertheless, they may also contain changes or alternative interpretations of some parts of the known storyline. The subject area of the original work is called fandom. The PAN organizers are providing unique author and fandom (topical) labels for all fanfiction pairs. The dataset has been derived from the corpus compiled in [3]. A detailed description of the dataset is given in [16]. As mentioned in the introduction, automatically extracted neural features should be invariant w.r.t. shifts in topic and/or sentiment. Ideally, LEVs should only contain information regarding the writing style of the authors. What is well-established in automatic AV is that the topic of a text generally matters. What is still not clear, however, is how stylometric or neural features are influenced/affected by the topic (i.e. fandom in this case). To increase the generalization capabilities of our model and to increase the model's resilience towards cross-topic fanfiction pairs we devised the following preprocessing strategies, as outlined in Sections 3.1 through 3.3.

Topic masking
Experiments show that considering a large set of token types can lead to significant overfitting effects. To overcome this, we reduced the vocabulary size for tokens as well as for characters by mapping all rare token/character types to a special unknown  (<UNK>) token. This is quite similar to the text distortion approach proposed in [21]. However, even when a rare/misspelled token is replaced by the <UNK> token, it can still be encoded by the character representation.

Sliding window with contextual prefix
Fanfiction frequently contains dialogues and quoted text. Sentence boundary detectors, therefore, tend to be very error prone and steadily fail to segment the data into appropriate sentence units. We decided to perform tokenization without strict sentence boundary detection and generated sentence-like units via a sliding window technique instead. An example that illustrates the procedure is shown in Fig. 2. We used overlapping windows to guarantee that semantically and grammatically linked neighboring tokens are located in the same unit. We also added a contextual prefix which is provided by the fandom labels. To initialize the prefix embeddings, we removed all non-ASCII characters, tokenized the fandom string and averaged the corresponding word embeddings. The final sliding window length (in tokens) is given by hop_length + overlapping_length + 1.

Data split and augmentation
To tune our model we split the datasets into a train and a dev set. Table 1 shows the resulting sizes. The size of the train set can then be increased synthetically by dissembling all predefined document pairs and re-sampling new same-author and different-author pairs in each epoch. We first removed all documents in the train set which also appear in the dev set. Afterwards, we reorganized the train set as described in Alg. 1 -3. Assuming the i-th author with i ∈ {1, . . . , N } contributes with N i fanfiction texts, we define a set is the corresponding fandom label. The objective is to obtain a new set D of re-sampled pairs, containing 5-tuples of the form (d (1) , d (2) , f (1) , f (2) , l), where d (1) , d (2) defines the sampled fanfiction pair, f (1) , f (2) are the corresponding fandom labels and l ∈ {0, 1} indicates whether the texts are written by the same author (l = 1) or by different end // Sample different-authors pair 14 if |G (1) | > 1 then 15 randomly draw (a (1) , f (1) , d (1) ) ∈ G (1) and (a (2) , f (2) , d (2) ) ∈ G (1) (1) and (a (2) , f (2) , d (2) ) ∈ A (2) 22 Besides the losses in Eqs. (3) and (16), we can also take into account the betweenauthor and within-author variations to validate the training progress of our model. Both, between-author and within-author variations can be characterized by determining the entropy w.r.t. the estimated covariance matrices B −1 and W −1 . It is well-known that entropy can function as a measure of uncertainty. For multivariate Gaussian densities, the analytic solution of the entropy is proportional to the determinant of the covariance matrix. From Eq. (5) and (6), we have Fig. 3 presents the entropy curves. As expected, during the training, the within-author variability decreased while the between-author variability increased. Fig. 4 shows the attention-heatmaps of two fanfiction excerpts. From a visual inspection of many of such heatmaps we made the following observations: In contrast to the Amazon reviews used in [4], fanfiction texts do not contain a lot of "easy-tovisualize" linguistic features such as spelling errors for example. The model focuses on different aspects. Similar to [4], the model rarely marked function words (e.g. articles,   pronouns, conjunctions). Surprisingly, punctuation marks like "..." seem to be less important than observed in [4]. In the first sentence of excerpt 1, the phrase "stopped and looked" is marked. In the second sentence, the word "look" of this phrase is repeated in the overlapping part but not marked anymore. Contrarily, repeated single words like "Absolutely" in excerpt 2 remain marked. It seems that our model is able to analyze how an author is using a word in a particular context.
Lastly, to keep the CPU memory requirements as low as possible on Tira [19], we fed every single test document separately and sequentially into the ensemble of trained models, resulting in a runtime of approximately 6 hours. This can, of course, be done batch-wise and in parallel for all models in the ensemble to reduce training time.

Conclusion and future work
We presented a new type of authorship verification (AV) system that combines neural feature extraction with statistical modeling. By recombining document-pairs after each training epoch, we significantly increased the heterogeneity of the train data. The proposed method achieved excellent overall performance scores, outperforming all other systems that participated in the PAN 2020 Authorship Verification Task, in both the small dataset challenge as well as the large dataset challenge. In AV there are many variabilities (such as topic, genre, text length, etc.) that negatively affect the system performance. Great opportunities for further gains can, thus, be expected by incorporating compensation techniques that deal with these aspects in future challenges.