{'fraction_non_alphanumeric': 0.0291061823004937, 'fraction_numerical': 0.010758234470562228, 'mean_word_length': 4.931818181818182, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
AraFacts: The First Large Arabic Dataset of Naturally Occurring Claims

We introduce AraFacts, the first large Arabic dataset of naturally occurring claims collected from 5 Arabic fact-checking websites, e.g., Fatabyyano and Misbar, and covering claims since 2016. Our dataset consists of 6,121 claims along with their factual labels and additional metadata, such as fact-checking article content, topical category, and links to posts or Web pages spreading the claim. Since the data is obtained from various fact-checking websites, we standardize the original claim labels to provide a unified label rating for all claims. Moreover, we provide revealing dataset statistics and motivate its use by suggesting possible research applications. The dataset is made publicly available for the research community.


Introduction
Fake news and misinformation are considered among the greatest threats to nations. The spread of fake news can cause manipulation in public opinion, which has adverse consequences to politics and journalism. Moreover, the recent COVID-19 pandemic revealed how medical misinformation could easily harm the health of people (Islam et al., 2020). Notable development has been made in automated fact-checking systems over the past years. However, one of the main limitations of Arabic automated fact-checking systems is the lack of Arabic datasets. Recently several Arabic fact-checking websites (e.g., Fatabyyano and Misbar) have emerged to help combat the spread of rumors and fake news, especially over social media. They constitute a valuable resource of naturallyoccurring fact-checked claims in the Arab world. The claims are annotated and verified by professional fact-checkers and journalists, making them a reliable source of information. While the verified information about those rumors is posted on those websites and their corresponding social media accounts, they are not gathered and unified as a dataset that can be used to develop automated systems. In this paper, we introduce AraFacts, the first large Arabic Fact-checking dataset of naturallyoccurring and professionally-verified claims. We crawl 6,222 claims along with their factual labels, description, and additional metadata, from 5 different fact-checking websites, and make them available to the research community in a normalized form. The contributions of our work are two-fold: • We extract and publicly share 1 about 6k claims with their original and normalized factual labels along with their metadata.
• We propose several research applications for which our dataset can be utilized.
The rest of the paper is organized as follows. Section 2 summarizes related work. Section 3 describes the data collection process. Section 4 provides data analysis. Section 5 outlines some research applications, and Section 6 concludes. Other datasets that focus on topic-specific claims were published recently. FakeCovid (Shahi and Nandini, 2020) dataset covers 5,182 multilingual fact-checked claims from 92 different fact-checking websites, all related to COVID-19. Moreover, PUBHEALTH dataset Kotonya and Toni (2020) consists of 11,832 claims public healthrelated claims attained from 8 different websites.

Related Work
Over the past few years, there has been an increased interest in task-specific Arabic applications leading to the release of several Arabic datasets. Most of these are built for tasks such as checkworthiness of tweets, evidence retrieval, and claim verification (Barrón-Cedeno et al., 2020;?).
Additionally, the Arabic News Stance dataset (ANS) (Khouja, 2020) supports claim verification and stance prediction. The dataset is generated using existing Arabic news titles from ANT corpus (Chouigui et al., 2017). The limitation with ANS is that it requires manual annotation to generate claims from news titles. Elmadany et al. (2020) address this limitation with the AraNews dataset. AraNews uses Arabic articles from online news data to automatically generate Arabic manipulated text. Baly et al. (2018) proposed a corpus consisting of 422 fact-checked Arabic claims and documents associated with the claims. The corpus supports multiple tasks such as stance detection and factchecking. AraFacts differs from their work in two ways; it covers a larger number of claims and all claims are professionally fact-checked.
While several datasets were constructed for different tasks that are related to fact-checking, to the best of our knowledge, AraFacts is the largest Arabic dataset that leverages emerging Arabic fact-checking websites as a source of annotated professionally-verified information.

Data Collection
In this section, we describe the process of selecting fact-checking websites, crawling the verification articles, and constructing the AraFacts dataset.

Fact-checking websites
We chose fact-checking websites that are either verified by the International Fact-Checking Network (IFCN) or popular in the Arab region. IFCN certification indicates that the website complies with IFCN's code of ethics. The following websites were selected as data sources: 1) Fatabyyano 4 is an IFCN-certified fact-checking organization that launched in 2016. Fatabyyano collaborates with Facebook as a third-party fact-checker to debunk fake-news, rumors, and conspiracy theories using Facebook's claim rating system. 5 2) FactuelAFP Arabic 6 is the Arabic bureau of the French press news service. Their team consists of journalists and fact-checkers. It aims to debunk false statements, videos, or images that appear online. It is certified by IFCN and collaborates with Facebook.
3) Misbar 7 is a popular independent Arabic fact-checking platform. It uses an 8-point claim rating system to label claims online. 4) Maharat-news fact-o-meter 8 is an IFCN-certified fact-checking website that focuses on investigating rumors online and in real-life using a 3-point claim rating system. 5) Verify-Sy 9 is a media platform that specializes in detecting and debunking false news and media by experienced journalists.

Data Extraction
From every fact-checking website, we crawled all fact-checking articles and extracted their claims, labels, and metadata. We also extracted the claim type using some indicative search keywords. The claim type identifies if the claim refers to textual information or if the claim is paired with visual information (such as a video or an image). For example; The claim MISB 2941 is referring to a fake image shown in Figure 2.
While parsing the content of the HTML pages, some challenges were faced. One challenge was that websites like Verify-Sy do not explicitly label the veracity of the claim. To address this challenge, we reviewed the website's editorial policy and inspected a subset of the claims manually. We concluded that all claims are, in fact, false. Moreover, not all metadata fields were available on all websites. Table 1 shows the number of crawled claims per website and a summary of available metadata.

Veracity and Category Normalization
Another faced challenge was the varying claim rating system adopted by the different websites. We proposed a normalized claim rating to achieve a standard rating for all claims in AraFacts. We also used a normalized rating for the categories of the claims.
To normalize the veracity label of the claim, we used the label normalization method presented by  (2020) with some variation. We set our own claim rating scheme consisting of four labels (False, Partly-False, Sarcasm, True) and mapped original labels to them. To do so, we referred to the source websites' methodology, manually inspected a subset of the claims, then finally did the mapping of 27 distinct original labels to four normalized labels. We adopted a similar mapping technique to map 35 distinct topical categories to 8 normalized categories: (Politics, News, Health, Social, Religion, General Sciences, Arts & Culture, and Other). More details on our claim and category normalization can be found in AraFacts repository. The normalization process was performed by two authors of this work independently; then, disagreements were discussed and resolved.

Dataset Construction
After extracting claim information and normalizing the veracity labels and categories, we assigned a unique ID to each claim. We finally merged the crawled information for each website into one database with the following schema: 1) Claim-ID: ID of the claim. 2) Claim: Text of the claim. 3) Source: Name of the fact-checking website from which the claim was crawled. 4) Description: Detailed description of the claim. 5) Source-label: The veracity label of the claim as it appears in the fact-checking website. 6) Normalized-label: Normalized claim label. 7) Source-category: Topical category of the claim as it appears in the fact-checking website. 8) Normalized-category: Normalized topical category of the claim. 9) Date: Article publication date. 10) Source URL: URL of the article. 11) Claim URLs: URLs to web pages spreading the claim. 12) Evidence URLs: URLs referenced by the fact-checker to justify their annotation. 13) Claim type: Indicates whether the claim refers to text, an image or a video.
Additionally, we extracted the content of the fact-checking article and included it in AraFacts. Figure 1 shows an example claim with some attributes.

Data Analysis
In this section, we perform further analysis on the collected claims. Table 1 gives an overall summary of the collected claims and some of the ex-  tracted metadata. In addition to the number of claims, for each website, we report the number of claims that have a refer to visual information (image or video), number of claims that have embedded claim URLs, average number of embedded URLs per claim, number of claims that have embedded evidence URLs, and average number of evidence URLs per claim. Figure 3 illustrates the distribution of the normalized labels over the normalized categories. Most claims are political or news, and false claims are expectedly dominating in all categories.
We also examined the original source domains of the claims from the embedded claim URLs. Table 2 lists the most frequent 5 claim URLs. It is worthwhile noting that social media platforms constitute the main source of claims and rumors in the dataset. Figure 4 shows the distribution of the claims over publication time. We notice that the majority of the claims were published in 2020; this is due to the  fact the most fact-checking websites in our dataset were launched in the past two years. Figure 5 shows the most frequent 100 words in the claim text. We notice a mix of political figures, religious entities, and country names among others.
Finally, we analyze the types of extracted claims. Figure 6 shows the frequency of the different types of claims (Text claims, Image claims, and Video claims) for each fact-checking website. Interestingly, the two most contributing fact-checking websites (Misbar and Fatabyyano) contain more visual claims than textual.

Example Use Cases
This section provides example use cases and research problems that can be supported by AraFacts.

Claim Verification
One of the main automated fake news detection tasks is claim verification. The task is defined as follows: given a claim, predict its veracity. The metadata can be used as features to classify the claim, such as claim source URLs. While we provide four normalized veracity labels of the claim, the original veracity labels can still be utilized following

Claim Retrieval
The claim retrieval task is defined as follows: given a claim, check whether this claim has already been checked. The importance of this problem stems from the fact that many posted claims over social media platforms are just repetitions of previously fact-checked old claims. Having a system that addresses this problem can help combat the spread of those circulating claims over social media.

Evidence Retrieval
The task is defined as follows: given a claim, retrieve evidential sentences that help in verifying or debunking the claim. As we provide the content of the fact-checking articles and the URLs of evidence pages of claims, AraFacts can be extended by further annotating evidence sentences from the evidence pages to support such task.

Image-based Fact-checking
The task is defined as follows; given a claim that is mainly about an image, predict the factuality of the claim using the image and claim. This task has been explored for English claims (Zlatkova et al., 2019), but it has not been studied in Arabic yet. While AraFacts does not include the images related to the claims, we indicate the claim type and provide the URL to the fact-checking articles where the images can be extracted and paired with the claims.

Conclusion
In this paper, we introduce AraFacts, the first large Arabic dataset of naturally-occurring claims covering about 6k claims published since 2016 and already annotated by professional fact-checkers from 5 different Arabic fact-checking websites over several topical categories. The dataset supports many research tasks such as claim verification, claim retrieval, also evidence retrieval. We made our dataset publicly available to the research community, and we plan for periodic crawling to augment the newly verified claims from existing and new fact-checking websites. We believe that AraFacts will serve as a valuable resource for future studies on fact-checking, such as fake image detection. Our future work will focus on evaluating and utilizing AraFacts for several fact-checking tasks to contribute the Arabic-based work in that domain.