{'fraction_non_alphanumeric': 0.023199175695506915, 'fraction_numerical': 0.004371311705748275, 'mean_word_length': 4.954266592303402, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Automated Fake News Detection in the Age of Digital Libraries

The transformation of printed media into the digital environment and the extensive use of social media have changed the concept of media literacy and people’s habits of news consumption. While online news is faster, easier, comparatively cheaper, and offers convenience in terms of people's access to information, it speeds up the dissemination of fake news. Due to the free production and consumption of large amounts of data, fact-checking systems powered by human efforts are not enough to question the credibility of the information provided, or to prevent its rapid dissemination like a virus. Libraries, long known as sources of trusted information, are facing challenges caused by misinformation as mentioned in studies about fake news and libraries. Considering that libraries are undergoing digitization processes all over the world and are providing digital media to their users, it is very likely that unverified digital content will be served by world’s libraries. The solution is to develop automated mechanisms that can check the credibility of digital content served in libraries without manual validation. For this purpose, we developed an automated fake news detection system based on Turkish digital news content. Our approach can be modified for any other language if there is labelled training material. This model can be integrated into libraries’ digital systems to label served news content as potentially fake whenever necessary, preventing uncontrolled falsehood dissemination via libraries.


INTRODUCTION
Collins dictionary which chose the term "fake news" as the "Word of the Year 2017," describes news as the actual and objective presentation of a current event, information, or situation that is published in newspapers and broadcast on radio, television, or online. 2 We are in an era where everything goes online, and news is not an exception. Many people today prefer to read their daily news online, because it is a cost-effective and convenient way to remain up to date. Although this convenience has lucrative benefits for society, it can also have harmful side effects. Having access to news from multiple sources, anytime, anywhere has become an irresistible part of our daily routines. However, some of these sources may provide unverified content which can easily be delivered right to your mobile device. Most importantly, potential fake news content delivered by these sources may mislead society and cause social disturbances such as triggering violence against ethnic minorities and refugees, causing unnecessary fear related to health issues, or even sometimes result in crisis, devastating riots and strikes.
Not having a steady definition compared to news, fake news is often defined according to the data used or the limited perspective of the study in the literature. For example; DiFranzo and Gloria-Garcia defined the fake news as "false news stories that are packaged and published as if they were genuine." 3 On the other hand, Guess  "lies presented as news." 4 A comprehensive list of 12 definitions can be found in Egelhofer and Lecheler. 5 In simplified terms, news which is created to deceive or mislead readers can be called fake news. However, the concept of fake news is a quite broad one that needs to be specified meticulously.
Fake news is created for many purposes and emerges in many different types. Having an interwoven structure, most of these types are shown in figure 1. Although, it is not easy to cluster these types into separate groups, they can be categorized according to the information quality or based on the intention as it is created to deceive deliberately or not, as Rashkin et al. did. 6 We propose the following classification where the two dimensions represent the potential impact and the speed of propagation. The four regions visualized are clustered according to their dangerousness. First of all, it should be noted that to order types of fake news in a stable precision is quite a challenging task. The variations within the field highly depend on dynamic factors such as timespan, actors, and echochamber effect. Hence, this figure should be considered as a clustering effort. There are possible intersecting areas of types within the regions. We will now give examples for two regions, "sr" and "SR." For example, the SR grouping shows characteristics of high-risk levels and fast dissemination. This includes varieties of fake news such as propaganda, manipulation, misinformation, hate news, provocative news, etc. We usually encounter this in the domain of politics. This kind of news may cause critical and nonrecoverable results in politics, the economy, etc., in a short period of time.
The rise of the term fake news itself can also be attributed to this kind of news. On the other hand, the relatively less severe group (sr) of fake news, comprising of satire, hoax, click-bait, etc., has low-risk levels and a slow speed of dissemination. A frequently used type of this group, click-bait, is a sensational headline or link that urges the reader to click on a post, link, article, image, or video. These kinds of news have a repetitive style. It can be said that readers become aware of falsehood after experiencing a few times. So, risk level is lower, and dissemination is slower.
Vosoughi et al. stated the assumption that "Falsehood diffuses significantly farther, faster, deeper, and more broadly than the truth." 7 So indeed, just one piece of fake news may affect many more people than thousands of true news items do because of the dramatic circulation of fake news.
In their recent survey about fake news, Zhou and Zafarani highlighted that fake news is a major concern for many different research disciplines especially information technologies. 8 Being a trusted source of information for a long time, libraries will play an important role in fighting against fake news problem. Kattimani et al. claims that the modern librarian must be equipped with necessary digital skills and tools to handle both printed collections and newly emerging digital resources. 9 Similarly, we foresee that digital libraries, which can be defined as collections of digital content licensed and maintained by libraries, can be a part of the solution as an authority service with a collective effort. Connaway et al. point to the key role of information professionals such as librarians, archivists, journalists, and information architects in helping society use the products and services related to news in a convenient way. 10 As libraries all over the world are transitioning into digital content delivery services, they should implement mechanisms to avoid fake and misleading content being disseminated through them under the guidance of information professionals.
To lay out proper future directions for the solution strategy, a clear understanding of interaction between library and information science (LIS) community and fake news must be addressed. Sullivan states that the LIS community has been affected deeply in the aftermath of the 2016 US presidential elections. 11 Moreover, he quotes many other scientists, emphasizing libraries' and librarians' role in the fight against fake news. For example, Finley et al. say that libraries are the direct antithesis of fake news, the American Library Association (ALA) called fake news an anathema to the ethics of librarianship in 2017, Rochlin emphasizes the role of librarians in this fight, and talks about the need to adopt fake news as a central concern in librarianship and many other researchers name librarians in the front lines of the fight against fake news. 12 Today, the struggle to detect fake news and prevent their spread is so popular that competitions are being organized (e.g., http://www.fakenewschallenge.org/) and conferences are being held (e.g., Bobcatsss 2020). The struggle against fake news can be classified under three main venues: • Reader awareness • Fact-checking organizations and websites • Automated detection systems The first item requires awareness of individuals against fake news and a collective conscience within the society against spreading fake news. (International Federation of Library Associations) infographic which contains eight steps to spot fake news. The RADAR framework and the Currency, Relevance, Authority, Accuracy, and Purpose (CRAAP) test are some of the efforts trying to increase reader-awareness of fake news. 14 Unfortunately, due to the nature of fake news and the clever way they are created triggering people's hunger to spread sensational information, it is very difficult to achieve full control via this strategy. Some studies explicitly showed that humans are prone to get confused when it comes to spotting lies or deciding whether a news item is fake or not. 15 Furthermore, people often overlook facts that conflict with their current belief, especially in politics and controversial social issues. 16 The second strategy focuses on third-party manually driven systems for checking and labelling content as fake or valid. Recently, we have seen many examples of offline and online organizations trying to work according to this strategy, such as a growing body of fact-checking organizations, start-ups (Storyzy, Factmata, etc.), and other projects with similar purposes. 17 Unfortunately, these manually powered systems cannot cope with the huge amounts of digital content being steadily produced. Therefore, they focus only on a subset of digital content that they classify as having higher priority. Even for this subset of content, their reaction speed is much slower than the fake information's spread speed. Therefore, automated and verified systems emerge as an inevitable last option.
The third strategy offers automated fact-checking systems, which once trained, can deliver content labelling at unprecedented speeds. Today, many researchers are researching automated solutions and building models with different methodologies. 18 Notwithstanding the latest studies, there is still a lot to do in the realm of automated fake news detection. Automated fact-checking systems will be detailed in the rest of the paper.
Thanks to the internet, the collections of digital content served by digital libraries can be accessed by a great number of users without distance and time limits. Therefore, we propose a solution to the problem by positioning digital libraries as automated fact-checking services, which label digital news content as fake or valid as soon as or before it is served through library systems. The main reason we associate this approach with digital libraries is their access to a wide variety of digital content which can be used to train the proposed mathematical models, as well as their role in the society as the publisher of trusted information. To this end, we develop a mathematical model that is trained using existing news content served by digital libraries, and capable of labelling news content as fake or valid with unprecedented accuracy. The proposed solution uses machine learning techniques with an optimized set of extracted features and annotated labels of existing digital news content. Our study mainly contributes (a) a new set of features highly applicable for agglutinative languages, (b) the first hybrid model combining a lexicon/dictionarybased approach with machine learning methods to detect fake news, and (c) a benchmark dataset prepared in Turkish for fake news detection.

LITERATURE REVIEW
Contemporary studies have indicated that social, economic, and political events in recent years, especially after the 2016 US presidential elections, are increasingly associated with the concept of fake news. 19 Since then, fake news has begun to be used as a tool in many domains. On the other hand, researchers motivated by finding automated solutions started to make use of machine learning, deep learning, hybrid models, and other methodologies for their solutions.
Although computational deception detection studies applying NLP (Natural Language Processing) operations are not new, textual deception in the context of text-based news is a new topic for the field of journalism. 20 Accordingly, we believe that there is a hidden body language of news text, which has linguistic clues indicating whether the news is fake or not. Thus, lexical, syntactic, semantic, and rhetorical analysis when used with machine learning and deep learning techniques offers encouraging directions.
The textual deception spread over a wide spectrum and the studies have utilized many different techniques. There are some prominent studies which took the problem as a binary classification problem utilizing linguistic clues. 21 Although it is still early to say the linguistic characteristics of fake news are fully understood, research into fake-news detection in English-language texts is relatively advanced compared to that in other languages. In contrast, agglutinative languages such as Turkish have been little researched when it comes to fake news detection. Agglutinative languages enable the construction of words by adding various morphemes, which means that words that are not practically in use may exist theoretically. For example, "gerek-siz-leş-tir-ebilecek-leri-miz-den-dir," is a theoretically possible word that means "it is one of the things that we will be able to make redundant," but it is not a practical one.
Shu et al. classified the models for the detection of fake news in their study. 22 According to this study, the automated approaches can focus on four types of attributes to detect fake news: knowledge based, style based, stance based, or propagation based. Among these, it can be said that the most useful approaches are the ones which focus on the textual news content. The textual content can be studied by an automated process to extract features that can be very helpful in classifying content as fake or valid.
Many scholars have tried to build models for automatic detection and prediction of fake news using machine learning algorithms, deep learning algorithms, and other techniques. These scholars approach the detection of fake news from many different perspectives and domains. For example, in one of the studies, scientific news and conspiracy news were used. 23 In Shu et al.'s study based on credibility of news, the headlines were used to determine whether the article was clickbait or not. In another study, Reis et al. worked on Buzzfeed articles linked to the 2016 US election using machine learning techniques with a supervised learning approach. 24 Studies which try to detect satire and sarcasm can be attributed to subcategories of fake news detection. 25 Our observation, in line with the general view, is that satire is not always recognizable and can be misunderstood for real news. 26 For this reason, we included satirical news in our dataset. It should be noted that although satire or sarcasm can be classified by automated detection systems, experts should still evaluate the results of the classification.
While some scholars used specific models focusing on unique characteristics, some others such as Ruchansky et al. proposed hybrid deep models for fake news detection making use of multiple kinds of features such as temporal engagement between users and news articles over time and generated a labelling methodology based on those features. 27 In related studies, many features such as automatic extracted features, hand-crafted features, social features, network information, visual features, and some others such as psycholinguistic features, are applied by researchers. 28 In this work, we focused on news content features, however the social context features can also be adapted using different tiers such as user activity patterns, analysis of user interaction, profile metadata, social network/graph analysis etc. to extract features. We also have some of these features in our data but not having ground truth quantitatively, we avoided using these features.

METHODOLOGY
In this section, we present our motivation for this work which we visualized in a framework and named Global Library and Information Science (GLIS_1.0). Subsequently, we discuss the construction of the automated detection system as the key element of the GLIS_1.0 framework. We explain the framework, model, dataset, features, and the techniques used in this section.

Framework
The main structure of the proposed framework is shown in figure 2. This framework consists of highly cohesive but flexible layers. Fact-checking organizations should also benefit from the output of ADSs, as instead of manually checking heaps of news content, they could now focus on news labeled as potentially fake by an ADS. Through GLIS, ADSs make the life of fact-checking organizations and digital libraries much easier, all the while increasing the quality of news served to the public.
Considering this is a high-level overview of a structure given in figure 2, there may be many other components, mechanisms, or layers, but the key elements of this structure are automated detection systems and the digital libraries. A critical approach to this framework can be why we need such an authority mechanism. The answer will be quite simple, technological progress is not the only solution. On the contrary, tech giants have already been subject to regulatory scrutiny for how they handle personal information. 29 Also, their policy related to political ads has been questioned. Furthermore, they are often blamed for failing to fight fake news. Indeed, there is an urgent need for a global action more than ever. Digital libraries are much more than a technological advancement. Hence, they should be considered as institutions or services which can be a great authority service to provide news to society since the printed media disappears day by day.
The threats caused by fake news are real and dangerous, but only recently have researchers from different disciplines been trying to find possible solutions such as educational, technological, regulatory, or political. Digital librarianship can be the intersection of all these solutions for promoting information/media literacy. Hence, digital librarianship will make use of many automated detection systems (ADS) to serve qualified news. In the following section, we discuss ADS in detail.

Model
An overview of our model of automated detection system solution which is very critical for the framework is shown in figure 3. Our fake news detection model consists of two phases. First is the Language Model/Lexicon Generation and the second is Machine Learning Integration. In this work, we used machine learning algorithms via supervised learning techniques which learn from labeled news data (training) and helps us to predict outcomes for unforeseen news data (test).

Dataset
We collected our data from three sources: • The primary source is the GDELT (Global Database of Events, Language and Tone) Project (https://www.gdeltproject.org/), a massive global news media archive offering free access to news text metadata for researchers worldwide. It can almost be considered a digital library of news in its own right. However, GDELT does not provide the actual news text and only serves processed metadata along with the URL of the news item. GDELT normally does not check for the validity of any news items. However, we have only used news from approved news agencies and completely ignored news from local and lesser-known sources to maximize the validity of the news we have automatically obtained through GDELT. Moreover, we have post-processed the obtained texts by cross-validating with teyit.org data to clean any potential fake news obtained through GDELT links.  news item, they tag them as fake, true, or uncertain. We used their results to automatically download and label each news text. • Lastly, our team collected manually curated and verified fake and valid news obtained from various online sources and named it as MVN (Manually Verified News). This set includes fake and valid news that we have manually accumulated in time during our studies and that were not overlapping with the news obtained from GDELT and teyit.org sources.
We named our dataset TRFN. In Phase 2, the data is very similar to the one we used in Phase 1. However, to see the effectiveness of model, we made modifications to exclude old news before 2017 and added new items from 2019. The news in our dataset span a time frame between 2017-2019 and are uniformly distributed. Table 1 outlines the dataset statistics, namely where the news text comes from, its class (fake or valid), the amount of distinct texts and the corresponding data collection method. It can be seen from the table that most of our valid news come from the GDELT source, whereas teyit.org, a fact-checking organization, contributes only fake news. All news items were processed through Zemberek (http://code.google.com/p/zemberek), the Turkish NLP engine for extracting different morphological properties of words within texts. After this processing phase, all obtained features were converted into tabular format and made available for future studies. This dataset is now available for scholarly studies upon request.
In a study of this nature, the verifiability of the data used is important. As we have already mentioned, most of the data we used comes from verified sources such as mainstream news agencies accessed through GDELT and teyit.org archives which are verified by teyit.org staff. All data used in training the mathematical models which are to be explained in the rest of the paper are either directly or indirectly verified.
Another important issue was generalizability of the dataset, which determines whether the results of the study are only applicable to specific domains or to all available domains. Although focusing on a specific news domain would clearly improve our accuracies, we preferred to work in the general domain and included news from all specific domains. The distribution of domains in our dataset is visualized in figure 4. This distribution closely matches the distribution one would experience reading daily news in Turkey. Hence, we have no domain specific bias in our training dataset. Zemberek developers (http://zembereknlp.blogspot.com/2006/11/kelime-istatistikleri.html) to find the most common words in Turkish experimented with over five million words is compatible with most common words in our corpus. This evidence can be attributed to representability of our dataset.
The last issue worth discussing is the imbalanced nature of the dataset. An imbalanced dataset occurs in a binary classification study when the frequency of one of the classes dominates the frequency of the other class in the dataset. In our dataset, the amount of fake news is highly surpassed by the amount of valid news. This generally results in difficulties in applying conventional machine learning methods to the dataset. However, it is a frequently observed phenomenon due to the disparity of variable classes in these kinds of problems in real world. To avoid potential problems due to the imbalanced nature of the dataset, we used SMOTE (Synthetic Minority Over-sampling Technique) which is an over-sampling method. 30 It creates synthetic samples of the minority class that are relatively close in the feature space to the existing observations of the minority class.

Features
In this study, we discarded some features because of their relatively low impact on overall performance during the exploratory data analysis and subsequently in the training phase. The most effective features we decided on are shown in table 2. Spell errors per sentences ComplexityScore The score of the complexity/readibility of the news Source

Labels
The URL or identifier of the news MainCategory The category of the news NewsSite The unique address of the news The language model features nRootScore and nRawScore are features that we have borrowed from our earlier study on fake news detection. 31 In that study, we focused on constructing a fake news dictionary/lexicon based on different morphological segments of the words used in news texts. These two scores were found to be the most successful ones in determining the fakeness/validity of a news text, one considering the raw form of the words, the other considering the root form.
The extracted features are ComplexityScore and SpellErrorScore. ComplexityScore basically represents the readability of the text. Studies for determining a good readability metric exist for the Turkish language. 32 We used a modified version of the Gunnig-Fog metric, which is based on word length and sentence length. 33 Since Turkish is an agglutinative language, we used word length instead of using the syllable count. We also made some modifications to normalize the scores. The average number of syllables per word syllable in Turkish is 2.6, so we defined a word as a long word if it has more than 9 letters. 34 For a given news text T, the Complexity Score (CS) can be computed by equation 1.
(1) = ( + * 100 10 ) The second Extracted Feature is SpellErrorScore. We foresee that there may be many more errors in fake news than in valid news. We calculated the spell error counts making use of Turkish (2)

= ( )
Finally, we included the metadata categories Source, MainCategory, and NewsSite as additional identifiers for the learning process.
Then, we combined features extracted from text representation techniques with the features shown in table 2 and trained the model with different classifiers. For text representation, we followed two directions for the experiments. First, we converted text into structured features with Bag of Words (BOW) approach in which text data is represented as the multiset of its words. Second, we experimented with N-grams which represents the sequence of n words, in other words splitting text into chunks of size N-words.
In the (BOW) model, documents in TRFN are represented as a collection of words, ignoring grammar and even word order, but preserving multiplicity. In a classic BOW approach, each document can be represented as a fixed-length vector with length equal to the vocabulary size. This means each dimension of this vector corresponds to the occurrence of a word in a news item. We customized the generic approach by reducing variable-length documents to fixed-length vectors to be able to use with varying lengths with many machine learning models. Because we ignore the word order, we reduced fixed length of counts as histograms as seen in figure 5. Assuming N is the number of news documents and W is the number of possible words in the corpus, it should be noted that in N*W count matrix, N is generally large but infrequent, because we have many news documents, but most words do not occur in any given document causing rareness of a term/word which is a drawback for the approach. Therefore, we modified the model to compensate the rarity problem by weighting the terms using TF-IDF measure which evaluates how important a word is to a document in a collection.
The other technique we used, N-gram model is the generic term for a string of words in computational linguistics, and it is extensively used in text mining and NLP tasks. The prefixes that replace the n-part indicate the number of consecutive words in the string. So, a unigram is referred to one word, a bigram is two words, and an n-gram is n words.

EXPERIMENTAL RESULTS AND DISCUSSION
In this section, the experimental process and the results are presented. All experiments are performed using the Scikit-learn library. To evaluate the performance of the model and proposed features we employed the precision, recall, F1 score (the harmonic mean of the precision and recall), and accuracy metrics. We did many experiments using different combinations of features. To be effective, a classifier should be able to correctly classify previously unseen data. To this end, we tuned the parameter values for all the classification models used. Then, models were trained and evaluated on TRFN dataset using 10-fold cross-validation.
In table 3, we present the ultimate best scores of the proposed model. The results are highly motivating to exemplify how useful automated detection systems can be as a key component of the integrated solution framework in figure 2. We compared the algorithms with three ultimate feature sets for having respectively consistent results to the other feature set combinations. Set1 stands for bigram+FOpt (Optimized Features), Set2 stands for BOWModified+ FOpt and Set3 stands for unigram+bigram+FOpt. The results show that there is a relative consistency in terms of performance across the models. In almost all models, the combination of unigram+bigram and optimized features sets (FOpt) gives better results than the other combinations. The ExtraTree Classifier model is chosen as the best due to its higher performance. This model is also known as Extremely Randomized Trees Classifier which is a type of ensemble learning technique aggregating the results of multiple decision trees collected in a "forest" to output its classification result. It is very similar to Random Forest Classifier and only differs in the manner of construction of the decision trees. So, we can also see closer results between these two classifiers. Every ADS in GLIS_1.0 framework may use its own way to detect fake news. The open source ADS may improve with feedbacks. Hybrid models and other techniques such as neural networks with deep learning methodology can also be used according to the data, language of news and the news features related with both social context and news content.

CONCLUSION AND FUTURE WORK
In this study we presented a novel framework which offers a practical architecture of an integrated system for identifying fake news. We have tried to illustrate how digital libraries can be a service authority to promote media literacy and fight against fake news. Because librarians are trained to critically analyze information sources, their contributions to our proposed model are critical. Accordingly, we see this work as an encouraging effort for the next collaborative studies among the communities of LIS and CS (computer science).
We think that there is an immediate need for LIS professionals to participate and contribute to automated solutions that can help detecting inaccurate and unverified information. In the same manner, we believe the collaboration of LIS professionals, computer scientists, fact-checking organizations, and pioneering technology platforms is the key to provide qualified news within a real-time framework to promote information literacy. Moreover, we put the reader at the core of the framework as the feed reader position while consuming news.
In terms of automated detection systems, we proposed a fake news detection model integration of dictionary-based approach and machine learning techniques offering optimized feature sets applicable to agglutinative languages. We comparatively analyzed the findings with several classification models. We demonstrated that machine learning algorithms when used together with dictionary-based findings yield high scores both for precision and recall.
Consequently, we believe once operational in the field, proposed workflow can be extended in the future to support other news elements such as photographs and videos. With the help of Social Network Analysis (SNA) it may be possible to stop or slow down the spread of fake news as it emerges. During all the experiments we did, this work also highlighted several tasks as future research directions such as: • The studies can be deepened to mathematically categorize the fake news types and the dissemination characteristics of each type can be analyzed. • The workflow has the potential to provide an automated verification platform for all news content existing in digital libraries to promote media literacy.