{'fraction_non_alphanumeric': 0.03744728553888859, 'fraction_numerical': 0.01782949874552928, 'mean_word_length': 4.769479519556514, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Language-Based Approach to Fake News Detection Through Interpretable Features and BRNN

‘Fake news’ – succinctly defined as false or misleading information masquerading as legitimate news – is a ubiquitous phenomenon and its dissemination weakens the fact-based reporting of the established news industry, making it harder for political actors, authorities, media and citizens to obtain a reliable picture. State-of-the art language-based approaches to fake news detection that reach high classification accuracy typically rely on black box models based on word embeddings. At the same time, there are increasing calls for moving away from black-box models towards white-box (explainable) models for critical industries such as healthcare, finances, military and news industry. In this paper we performed a series of experiments where bi-directional recurrent neural network classification models were trained on interpretable features derived from multi-disciplinary integrated approaches to language. We apply our approach to two benchmark datasets. We demonstrate that our approach is promising as it achieves similar results on these two datasets as the best performing black box models reported in the literature. In a second step we report on ablation experiments geared towards assessing the relative importance of the human-interpretable features in distinguishing fake news from real news.


Introduction
The topic of 'disinformation' -an umbrella term used to encompass a wide range of types of information disorder, "including 'fake news', rumors, deliberately factually incorrect information, inadvertently factually incorrect information, politically slanted information, and 'hyperpartisan' news" (Tucker et al., 2018) -is attracting more and more attention. This reflects a deeper concern that the prevalence of disinformation leads to an increased political polarization, decreases trust in public institutions, and undermines democracy. For example, the spread of 'fake news' -concisely defined as intentionally false information masquerading as genuine news -for financial and political gains had a potential impact on the contentious Brexit referendum or 2016 U.S. presidential elections (Allcott and Gentzkow, 2017;Ward, 2018). Against this background, it is hardly surprising that there has been an increased interest in the development of methods, measures and computational tools that efficiently and effectively detect disinformation using machine learning and deep learning techniques. Among different approaches to fake news detection, language-based approaches have emerged as promising (for more details, see Section 2). Here the term 'language-based' is used in a broad sense to include a variety of approaches, such as those that employ traditional linguistic features, readibility features, style-based features, discourse and rhetorical features or those that draw on word embedding techniques. The latter have proven to be particularly successful in detecting fake news. Despite their success, however, their detection is based on latent features that are not human interpretable and thus cannot explain why a piece of news was detected as fake news. As recently pointed out by , white-box (explainable) approaches to fake news detection are desirable, since model-derived explanations can (1) provide valuable insights originally hidden to different stakeholders, such as policy makers, professional journalists and citizens and (2) can contribute to further improvement of fake news detection systems. This paper seeks to respond to recent calls for more explainable (white-box) approaches to fake news detection by performing a series of experiments where bi-directional recurrent neural network classifiers were trained on interpretable features derived from multi-disciplinary integrated approaches to language. The data come from two benchmark datasets and fake news detection is formulated as a binary classification and as a multiclass classification tasks correspondingly. The results of our experiments are promising, as our classification models achieve similar performance as the best-performing black box models reported in the literature. In a second step we report on ablation experiments geared towards assessing the relative importance of the human-interpretable features in distinguishing fake news from real news. The remainder of the paper is organized as follows: After a concise overview of related work in Section 2, Section 3 introduces the two data sets, Section 4 describes our approach to automated text analysis and six groups of language features used in the paper, Section 5 describes the model architecture, the training procedure and the method used to assess the relative feature importance. Sections 6 presents and discusses the main results and concluding remarks follow in Section 7.

Related Work
Here we provide a concise overview of recent approaches geared towards fake news detection that employ machine learning and deep learning techniques and we focus in particular on language-based approaches that are most pertinent to the purposes of this paper (for a more systematic and comprehensive overviews, see recent reviews and surveys by Shu et al. (2018), Oshikawa et al. (2020), Zhang and Ghorbani (2020) and Zhou and Zafarani (2020). Fake news detection is most often formulated as a binary classification task. However, categorizing all the news into two classes (fake vs real) is not the only conceivable way, since there are cases where the news is partially real and partially fake. A common practice is to add more classes distinguishing between several degrees of truthfulness and thus formulating fake news detection as a multi-class classification task. As will become evident later in this paper, we apply our approach to both scenarios. Three approaches to fake news detection frequently described in the literature are: (1) knowledge-based fake news detection (commonly using techniques from information retrieval to determine the veracity/truthfulness of news), (2) language-based fake news detection (drawing on traditional linguistic, style-related, readability or rhetorical features or using word embedding methods to distinguish between fake and real news) and (3) propagation-based fake news detection (typically using network analyses to determine the credibility of news sources at various stages, being created, published online and their spread via social media). Compared to knowledge-based and propagation-based approaches, language-based approaches are advantageous for several reasons, including: (1) they enable near real-time feedback (proactive rather than retroactive), i.e. they are not restricted to being applied only a posteriori (Potthast et al., 2017) and (2) they are scalable. A guiding assumption of language-based approaches is that there are statistical regularities inherent in natural languages and distributional patterns of language use indicative of fake news that are not consciously accessible to fake news creators. Space limitations prevent us from going into further details (but see reviews and survey cited above). In what follows, we will zoom in on previous studies on fake news detection conducted on the bases of the publicly available benchmark datasets used in the corpus study: the ISOT dataset, an 'entire article' dataset comprising 20k+ real and fake news texts (Ahmed et al., 2018), and the LIAR dataset, a 'claims dataset' comprising 12k+ real-world short statements collected from a variety of online sources (Wang, 2017) (see section 3 for details). Upon introduction of the ISOT dataset, (Ahmed et al., 2018) report on the results of experiments using n-gram features with two different features extraction techniques -Term Frequency (TF) and Term Frequency-Inverted Document Frequency (TF-IDF) -and six different machine learning techniques -Stochastic Gradient Descent, Support Vector Machines, Linear Support Vector Machines (LSVM), K-Nearest Neighbour and Decision Trees. Their best-performing model reached a classification accuracy of 92% using TF-IDF for feature extraction and an LSVM classifier, showing that real and fake news can be discriminated with high accuracy on the basis of the use of multiword sequences. However, subsequent studies have demonstrated that classification accuracy on this dataset can be pushed even higher -beyond the 99% accuracy mark -through the employment of deep neural networks trained on word embedding vectors: (Kula et al., 2020) reported classification accuracy between 95.04% and 99.86% using an LSTM neural network trained on different word em-beddings (glove, news, Twitter, crawl) implemented in the Flair NLP framework (Akbik et al., 2019). Goldani et al., (2020) achieved a classification accuracy of 99.8% using a non-static capsule network and 'glove.6B.300d' word embeddings (Pennington et al., 2014). While the dataset ISOT involves a binary classification (fake vs. real), the LIAR dataset presents a six-way multiclass classification problem, where individual claims statement was evaluated for its truthfulness and received a much more finegrained veracity label. In the experiments presented upon publication of the LIAR dataset,  provided several benchmarks based on several shallow learning classifiers (e.g. logistic regression and support vector machines) trained on n-gram features and deep learning classifiers (bi-directional long short-term memory and convolutional neural networks architectures) using pre-trained 300-dimensional word2vec embeddings from Google News (Mikolov et al., 2013). The latter reached a classification accuracy task of up to 27%. Incorporating available meta-data about the subject, speaker and context raised classification accuracy to 27.4%. Subsequent studies have shown that the classification accuracy on the LIAR set can be further increased to just over 45% by more complex hybrid models that integrate the linguistic information with speaker profiles into an attention based LSTM model (Long, 2017), by supplementing the data with verdict reports written by annotators (Karimi et al., 2018) or by replacing the credibility history in LIAR with a larger credibility source (Kirilin and Strube, 2018). Importantly, however, all state-of-the-art models designed to detect the veracity of a news article or claim exploit the information contained in high-dimensional word embeddings that are uninterpretable to humans, thereby severely limiting our ability to understand 'why' a given claim or news article was predicted to be fake or real.

Data
The experiments were conducted on two recently released datasets for fake news detection, the ISOT dataset compiled by the Information Security and Object Technology research lab (Ahmed et al., 2018) and the LIAR dataset introduced in (Wang, 2017). The datasets were selected based on their complementary attributes in terms of text types (full articles with average length of about 400 words vs. short statements with an average length of just under 20 words) and the granularity of the veracity labels (binary labels based on source selection and six-way classification based on ratings by politifac.com editors). Both datasets are sufficiently large for training deep models. The ISOT dataset consists of 40,000+ real and fake news articles collected from real-world sources between 2016 and 2017. The real (truthful) news articles were obtained by crawling articles from Reuters.com. The fake news articles were collected from unreliable websites that were flagged by politifact.com, a fact-checking organization in the USA, and Wikipedia. The ISOT dataset contains articles on a variety of topics with a focus on political and world news topics (see Table 2). For each article the following information is provided: article title, text, type (topic) and publication date. Close inspection of the dataset revealed that all and only instances of real news were introduced by the words "WASHINGTON (Reuters)", indicating the place and name of the news agency that has provided the news article. To prevent our models from capitalizing on this information, all instances of this string were deleted. We also checked for and removed all duplicates in the dataset (N = 6251). Table 2 presents the distribution of articles across news types (real/fake) and topics before and after deduplication (original/cleaned). The dataset was split in training, development, and testing sets using a 80/10/10 split. The LIAR dataset is a recent benchmark dataset for fake news detection that in includes 12,836 real-world short statements collected from a variety of online sourcesincluding Facebook posts, tweets, news releases, TV/radio interviews, campaign speeches, TV ads and debates -on a range of topics -including economy, healthcare, taxes, federal-budget, education, jobs, state budget, candidates-biography, elections, and immigration. Each statement was labeled by an editor from politifact.com on a six-level ordinal scale of truthfulness ranging from "True", for completely accurate statements, to "Pants on Fire" (from the taunt "Liar, liar, pants on fire") for false and ludicrous claims. The distribution of the six labels is relatively well-balanced: with the exception of 1,050 instances of the 'pants-fire' category, the instances for all other labels range from 2,063 to 2,638. The LIAR set further includes a rich set of meta-data for each speaker including party affiliation, current job and home state. The statements in the dataset are also fairly balanced across the two major political par-ties of the US -democrats and republicans -and also contain a significant amount of posts from online social media. The dataset is distributed into training, validation and testing sets in a 80/10/10 manner.

Automated Text Analysis
The raw texts from the two datasets were automatically analyzed using CoCoGen, a computational tool that implements a sliding window technique to calculate within-text distributions of feature scores (see recently published papers that use this tool, (Ströbel et al., 2018;Kerz et al., 2020b;Kerz et al., 2020a). In contrast to the standard approach implemented in other tools for automated text analysis that rely on aggregate scores representing the average value of a feature in a text, the sliding-window approach generates a series of measurements representing the 'local' distributions of scores. A sliding window can be conceived of as a window of size ws, which is defined by the number of sentences it contains. The window is moved across a text sentence-by-sentence, computing one value per window for a given feature. The series of measurements faithfully captures a typically non-uniform distribution of features within a text and is referred here to as a 'contour'. 1 To compute the value of a given feature in a given window m (w(m)), a measurement function is called for each sentence in the window and returns a fraction (wn m /wd m ). CoCoGen uses the Stanford CoreNLP suite  for performing tokenization, sentence splitting, part-of-speech tagging, lemmatization and syntactic parsing (Probabilistic Context Free Grammar Parser (Klein and Manning, 2003)). In its current version, CoCoGen supports a total 154 of features that fall into six categories: (1) features of syntactic complexity (N=19), (2) features of lexical density, sophistication and variation (N=12), (3) information-theoretic features (N=3), (4) register-based n-gram frequency features (N=25), (5) LIWC-style (Linguistic Inquiry and Word Count) features (N=61) and (6) Word-Prevalence measures (N=36). A brief overview of the features and their short descriptions are provided in Table 4 in the Appendix. The inclusion of these features 2 is motivated by contemporary language and cognitive sciences characterized by an integrated, multi-method, and transdisciplinary approach needed to advance our understanding of the human processing and learning mechanisms (Christiansen and Chater, 2017). The first three sets of features are derived from the literature on language development showing that, in the course of their lifespan, humans learn to produce and understand complex syntactic structures, more sophisticated and diverse vocabulary and informationally denser language (see, e.g., Berman, 2007;Lu, 2010Lu, , 2012Hartshorne and Germine, 2015;Ehret and Szmrecsanyi, 2019). The fourth set of features is derived from research on language adaptation (Chang et al., 2012) and research that looks at language from the perspective of complex adaptive systems (Beckner et al., 2009;Christiansen and Chater, 2016) indicating that, based on accumulated language knowledge emerging from lifelong exposure to various types of language inputs, humans learn to adapt their language to meet the functional requirements of different communicative contexts. The features in set five are based on insights from many years of research conducted by Pennebaker and colleagues (Pennebaker et al., 2003;Tausczik and Pennebaker, 2010), showing that the words people use in their everyday life provide important psychological cues to their thought processes, emotional states, intentions, and motivations. And finally, the inclusion of features in group six is motivated by recent efforts to estimate of of how well words are known in the population through crowdsourcing and corpus-based techniques. An accumulating body of evidence shows that such word prevalence measures are good predictors of human perfomance on various language tasks (Brysbaert et al., 2019;Johns et al., 2020)

Classification Models
For the classification, we used Bi-directional Recurrent Neural Network (BRNN) classifiers with Gated Recurrent Unit (GRU) cells . BRNNs have been shown to outperform unidirectional RNNs in application areas ranging from acoustic modeling (Sak et al., 2014) to machine translation . Bi-directional neural network models have also been employed in previous studies on the two datasets investigated here Kula et al., 2020), making them well suited for purposes of comparison and, more specifically, for examining whether and to what extent a classifier trained on human-interpretable features can approximate the performance of a state-of-the-art classifier trained on word embeddings. Since the two datasets differ in terms of the availability of meta-data (ISOT: no meata-data, LIAR: rich information on subject, speaker and context) and with respect to the granularity at which truthfullness was assessed (ISOT: binary, LIAR: 6-way multiclass), the BRNN classifiers were adapted so as to take these differences into account. Figure 1 shows the architecture of models used in the present paper. X = (x 1 , x 2 , . . . , x n ) is the output from CoCoGen, which is a sequence of 154dimensional vectors. To integrate the context information, the words in the context description were mapped to 300-dimensional word embedding vectors using the dependency based word-embedding implemented in spaCy (Honnibal and Montani, 2017), represented by C = (c 1 , c 2 , . . . , c n ). Instead of one-hot encoding, we use word embeddings and BRNN to encode the context meta information here, as otherwise the feature vector for context information would result in 5075-dimensional sparse one-hot vectors. J = (j 1 , j 2 , . . . , j n ) is a sequence of word embeddings for the job title of the speaker of a given text, following the same reasoning as above. S = (s 1 , s 2 , . . . , j n ) and P = (p 1 , s 2 , . . . , p n ) are 70 and 25 dimensional on-hot vector for state information and party affiliation of the speaker. The structure of the classifier for ISOT dataset is shown in 1 on the left hand side in Figure 1. The lower part encircled by the dashed red line represents the recurrent network, where the CoCoGen output for a given text is fed into a 2-layer BRNN consisting of GRU cells with 200 hidden units in each layer. h 10 , h 20 represent the initial hidden states of the first and second layer of the BRNN respectively in the forward direction and h 10 , h 20 represent the initial hidden states of the first and second layer of BRNN respectively in the backward direction. h 2n and h 2n represent the last hidden states of the second layer of the BRNN in the forward and backward direction respectively. These layers are concatenated and passed through a feed-forward neural network, encircled by the blue dashed line in Figure 1. This network consists of three linear layers, whose output dimensions are 200, 100 and 2. Between layers 1 and 2 as well as between layers 2 and 3 we inserted a Batch Normalization (BN) layer, a Parametric ReLU (PReLU) activation function layer and a Dropout layer with a dropout rate of 0.5. A softmax layer is applied before the final outputŷ. For the LIAR dataset, we built three BRNN models: (1) a model using only the CoCoGen output (X), (2) a model using CoCoGen output and the context information (X + C) and (3) a model using CoCoGen output, the context information and the speaker profile, which comprises information about the job, the state and the party of the of a speaker (X + C + J + S + P). The structure of CoCoGen-only model is identical to model built for the ISOT dataset, with the exception that the output layer has a size of 6 instead of 2. In the CoCoGen + Context model shown in sub-figure 2 in Figure 1, the sequence vector X = (x 1 , x 2 , . . . , x n ) represents the CoCoGen output as described for the ISOT model above. BRNN blocks in sub-figure 2 has a same structure as the lower part of sub-figure 1, which is a 2-layer bidirectional RNN, whose output is a concatenation of the last hidden state of uppermost layer in forward and backward direction respectively. The BRNN on the left size in sub-figure 2 has a hidden state size of 200, while the BRNN on the right side has one of 10. The Feed-forward 1 block is identical to the Feed-forward part shown in sub-figure 1. Sub- figure 3 shows the structure of model making use of CoCoGen features + context + speaker profiles. S and P are one-hot encoded vectors described as above. They are squeezed to 10-dimensional vectors through a feed-forward neural network, Feed-forward 2, whose structure is shown in the lower right part of Figure 1. Feed-forward 2 consists of two linear layers, the output of which are 20 for Linear 1 and 10 for Linear 2 respectively. The BRNN for CoCoGen output and context are identical to the corresponding BRNN blocks mentioned above. The BRNN for job title information encoding has the same structure and hidden state size as BRNN for context. All output from BRNN blocks and Feed-forward 2 blocks are concatenated and fed into Feed-forward 1 block, whose structure is shown in the upper part of sub- figure 1 with the exception that linear layers have output size of 210, 105 and 5 respectively. Since the labels of the LIAR dataset are ordinal in nature, i.e. pants-fire < false < barely-true < half-true < mostly-true < true, the classification of instance in liar dataset can be treated as an ordinal classification problem. To adapt the neural network classifier to the ordinal classification task, we followed the NNRank approach described in (Cheng et al., 2008), which 1 2 3 Feed-forward 2 BRNN Feed-forward 1 Figure 1: Structure of the BRNN classifiers built for the ISOT and LIAR datasets: The structure in 1 represents the model architecture used for ISOT and LIAR that makes use of textual information only (all CoCoGen features). The structures in 2 and 3 represent the model extensions that incorporate contextual meta-data (C) and speaker profiles (J = job title, P = party affiliation, S = speaker).
is a generalization of ordinal perception learning in neural networks (Crammer and Singer, 2002) and outperforms a neural network classifier on several benchmark datasets. Instead of one-hot encoding of class labels and using softmax as the output layer of a neural network, in NNRank, a class label for class k is encoded as (y 1 , y 2 , . . . , y i , . . . , y C−1 ), in which y i = 1 for i ≤ k and y i = 0 otherwise, where C is the number of classes. For the output layer, a sigmoid function was used. For prediction, the output of the neural network (o 1 , o 2 , . . . , o C−1 ) is scanned from left to right. It stops after encountering o i , which is the first element of the output vector that is smaller than a threshold T (e.g. 0.5), or when there is no element left to be scanned. The predicted class of the output vector is the index k of the last element, whose value is greater than or equal to T . Finally, for the purpose of comparison, we also recreated the convolutional neural network (CNN) model described in ). This CNN model consists of filters of size 2, 3 and 4. Each size has 128 filters with a max-pooling operation being performed on each output filter. The result of the max-pooling was fed into a feed-forward neural network for the classification. As an additional baseline, we further built structurally equivalent BRNN classifiers based on sentence embeddings from Sentence-BERT (SBERT) (Reimers and Gurevych, 2019). 3 All models are implemented using PyTorch (Pytorch, 2019). For the BRNNs and the CNN that don't use the ordinal information cross entropy loss was used as a loss function: where c is the true class label of the current observation, C is the number of classes, (p(y 1 ), . . . , p(y C )) is a one-hot vector with p(y i ) = 1 i = c 0 otherwise andŶ = (p(ŷ 1 ), p(ŷ 2 ), . . . , p(ŷ C )) is the output vector of the softmax layer, which can be viewed as the predicted probabilities of the observed instance falling into to each of the classes. For training BRNNs using ordinal information binary cross entropy was used: in which c = (y 1 , y 2 , . . . , y N ), C = 14 is number of responses andŶ = (ŷ 1 ,ŷ 2 , . . . ,ŷ N ) is the output vector of the sigmoid layer rounded to closest integer. We tuned all hyperparameters on the validation set using a grid search over sets of optimizers S = {Adamax, Adagrad, RMSprop}, learning rates L = {0.01, 0.001, 0.0001} and normalization methods N = {Standardization, Min-max}. The optimal hyperparameter combinations are provided in Table 5 in the Appendix.
To determine the relative importance of the language features groups, we conducted feature ablation experiments. Classical forward or backward sequential selection algorithms that proceed by sequentially adding or discarding features require a quadratic number of model training and evaluation in order to obtain a feature ranking (Langley, 1994). In the context of neural network models, training a quadratic number of models can become prohibitive. To alleviate this problem, we used an adapted version of the iterative sensitivity-based pruning algorithm proposed by (Díaz-Villanueva et al., 2010). This algorithm ranks the features based on a 'sensitivity measure' (Moody, 1994;Utans and Moody, 1991) and removes the least relevant variables one at a time. The classifier is then retrained on the resulting subset and a new ranking is calculated over the remaining features. This process is repeated until all features are removed. In this fashion, rather than training n(n+1) 2 models required for sequential algorithms, the number of models trained is reduced to n m , where m is the number of features or feature groups that can be removed at each step. We report the results obtained after the removal of a single feature group at each step. At step t, a neural network model M t is trained on the training set. The training set at step t consists of instances with feature groups F t = {f 1 , f 2 , . . . , f Dt } where f 1 , . . . f Dt are the remaining feature groups at the current step, whose importance rank is to be determined. We define X t as the test set with feature set F t and X i t as the same dataset as X t except we set the i th feature f i of each instance within the dataset to its average. Furthermore, we define g(X) as the classification accuracy of M t,n for a dataset X. The sensitivity of a feature group f i at step t is obtained from: t ) The most important feature group at step t can be found by: Then we set the rank for feature fî: In the end, feature fî is dropped from F t and the corresponding columns in training and test dataset are also dropped simultaneously: This procedure is repeated, until |F t | = 1.

Results
The performance metrics of the classification models for both datasets (global accuracy, precision and recall) are presented in Table 1, along with comparisons with the results of previous studies (a extended version of the table with performance data of additional models is provided in the Appendix). The results of our BRNN classifiers trained on interpretable features are highly competitive with those obtained from state-of-the-art RNN, CNN and capsule networks that exploit word embeddings to represent textual contents. In fact, in both datasets, our classifiers match the performance of the best-performing models within half a percent: For ISOT, the CAPSULE-glove (Goldani et al., 2020) and LSTM-glove (Kula et al., 2020) both achieve an accuracy of 99.8%, while BRNN CoCoGen achieves 99.3%. Moreover, the BRNN CoCoGen model outperformed the LSTMs presented in Kula et al. (2020) that utilize three other word embeddings implemented in the Flair library (news, Twitter, crawl) by up to 4.3% and improved on the performance on the n-gram-based LSVM model by 7.3%. For the LIAR data set, the difference in classification accuracy between BRNN CoCoGen and the CNN utilizing 300-dimensional word embeddings trained on Google News presented in Wang (2017) amounts to 0.2%, when meta-data on context and speaker profiles is taken into account. Excluding all meta-data, the BRNN CoCoGen (ordered) model reached an accuracy of 27.7%, which is even slightly higher than the performance of the Bi-LSTM 300dim word2vec embeddings (Google news) model. Our CNN CoCoGen model achieved a classification accuracy of 25.6%, which is 1.4% below the performance of the corresponding CNN model presented in Wang (2017) , CNN 300-dim word2vec embeddings (Google News). Interestingly, however, this model suffered from a substantial drop in accuracy to 24.8%, once it was infused with contextual meta-data. In contrast, all BRNN CoCoGen models invariably benefited from the addition of any type of meta-data. While performance with the CAPSULE-glove networks presented in Goldani et al. (2020) is limited by their selective integration of meta-data, it is worth noting that the CNN CoCoGen model outperformed all their models without recourse to meta-data. Taken together these results present strong evidence that successful detection of fake news can be achieved without sacrificing transparency. It is also worth pointing out that approaching the fake news detection task as an ordinal classification problem had considerable effects on a classifiers performance. Specifically, we observed (1) that classification accuracy slightly increased by 0.6% relative to a unordered classification approach and (2) that classification behavior shifted from a bias towards recall to a bias towards precision. Furthermore, comparison of the confusion matrices of our classifiers revealed that changing to the ordinal classification approach had positive effects on the distribution of errors: The ordinal classification problem is monotonic, meaning that the further a misclassification is from the main diagonal of a confusion matrix, the more severe it is. The confusion matrix of the best-performing BRNN CoCoGen model shows that for five out of the six classes (pants-fire, false, half-true, mostly-true, true) the most frequent prediction was the true class and the number of misclassifications decreases with increasing distance to the true class. In contrast, in the case of the unordered classifiers, we observed that the extreme categories ('pants-fire' and 'true') were avoided and predictions to the intermediate categories were preferred, especially in classifiers without meta-data information (confusion matrices for all models are provided in the Appendix). To the best of the authors knowledge, current models on multi-class fake news detection do not concern with the order of labels (Oshikawa et al., 2018). Our results indicate that future work can benefit from taking an ordinal classification approach. The results of our feature ablation experiments revealed a similar rank order in feature importance in both datasets (detailed results can be found in Table 14 in the Appendix): In each case, classification performance was mainly driven by features from the groups Lexical, LIWC, Syntactic and register-based n-grams, and to a lesser extent by information theoretic and word-prevalence-based features. Specifically, Table 14 indicates that -in the casse of the ISOT dataset -dropping the features from the LIWC group results in the largest decrease in classification accuracy of 5.1% on the validation set, resulting in a drop in accuracy on the test set to 93.8%. Re-training the model without the LIWC features yields the new baseline of 99.1%, indicating that the remaining features contained enough information to allow the retrained model to compensate for the loss of the LIWC information. After the elimination of the next two most-important feature groups, the syntactic and lexical groups, the retrained model at iteration 3 is still able to achieve an accuracy on the validation set of 97.9%. However, after the drop of the n-gram feature group, classification accuracy on the drops to 76.3% (validation) and 76.2% (test), indicating that the lost information from the four top-feature groups cannot be compensated for by information from the remaining feature groups, i.e. information theoretic and word-prevalence-based features. In the case of the the LIAR dataset, the relative influence of the six feature groups is more even and the predictive power of the model (27.2% accuracy on the test set) appears to stem from exploiting information from all six feature groups. For a closer examination of how individual features within each feature-group distinguished between real and fake news, we derived standard scores by performing z-standardization on all indicators and determined the difference between mean standard scores of real and fake news (DeltaScore index i = Score index i, f ake news − Score index i, real news ) (a complete table with the DeltaScores for the top-20 features for both datasets is provided in the Appendix). Inspection  Kula et al., 2020;3 = Goldani et al., 2020;4 = Wang, 2017 of the Delta Scores revealed some interesting patterns. For example, real news articles and claims are characterized by (1) relatively higher lexical diversity (as measured by type-token ratio features), (2) stronger reliance of multiword sequences from the news and academic register (measured by registerbased n-gram frequency measures), (3) greater phrasal syntactic complexity (as measured, e.g., by the number of complex nominals per clause) and (4) more frequent use or word from particular domains, such as work, money, power or word classes, such as preposition and quantifiers. In contrast, fake news are characterized by (1) greater syntactic complexity (as measured by, e.g. by the number of clauses per sentence), (2) frequent use of multiword sequences form the domain of fiction, (3) higher lexical sophistication scores (as measured in terms of relatively infrequent words) and (4) a strong reliance on personal pronouns, adverbs and emotion words. While limitations of space preclude an in-depth discussion, these results demonstrate that the use of interpretable features can provide new insights and knowledge about the characteristics of fake news and explain "why" a piece of news was detected as fake news see  for a discussion of explainable fake news detection.

Conclusion and Future Work
In recent years, there is a growing recognition of the need to move away from black-box models towards white-box models for solving practical problems, in particular in the context of critical industries, including healthcare, criminal justice, and news (Rudin, 2019). This is due to the fact that human experts in a given application domain need both accurate but also understandable models (Loyola-Gonzalez, 2019). In this paper, we have made a contribution to this development in the domain of fake news detection. We have demonstrated that models trained on human interpretable features in combination with deep learning classifiers can compete with black box models based on word embeddings. In the future we intend to extend this work in two directions: First, we plan to apply our approach to fake news detection in German whose research still lags far behind that available for English. Second, we also plan to apply our approach to the detection of rumours and conspiracy theories to tackle and combat the ongoing Covid-19 infodemic.    Table 6: Evaluation results on the ISOT and LIAR datasets on the validation and test sets. Models indexed as "CoCoGen" comprise textual features only. Models with "+" are hybrid models with textual and meta-data. The labels "ordered" and "unordered" indicate whether an ordinal and nominal classification method was applied.  1  23  24  31  11  2  false  2  49  55  84  53  6  barely-true  3  36  62  80  27  4  half-true  1  38  58  108  55  5  mostly-true  1  19  41  104  74  2  true  3  19  43  71 66 6