{'fraction_non_alphanumeric': 0.032420091324200914, 'fraction_numerical': 0.024657534246575342, 'mean_word_length': 4.450435503940274, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Development of a Web-Scale Chinese Word N-gram Corpus with Parts of Speech Information

Web provides a large-scale corpus for researchers to study the language usages in real world. Developing a web-scale corpus needs not only a lot of computation resources, but also great efforts to handle the large variations in the web texts, such as character encoding in processing Chinese web texts. In this paper, we aim to develop a web-scale Chinese word N-gram corpus with parts of speech information called NTU PN-Gram corpus using the ClueWeb09 dataset. We focus on the character encoding and some Chinese-specific issues. The statistics about the dataset is reported. We will make the resulting corpus a public available resource to boost the Chinese language processing.


Introduction
In recent years, researchers often use web-scale texts to train their models to alleviate the data sparseness problem in statistical natural language processing. Empirical study showed that the use of web-scale resources result in more robust models (Bergsma et al., 2010). Although the web pages contain rich language usage phenomena, processing web-scale texts is not an easy task because it not only needs a lot of computation resources, but also has to filter out the noises in messy web texts. Instead of raw web pages, some search engine providers, such as Google and Microsoft, provide word N-gram corpora or Web N-gram language models for researchers (Wang et al., 2010;Liu et al., 2010;Brants & Franz, 2006). The Google N-gram corpus contains word N-grams and their counts extracted from trillion words of web pages. Microsoft N-gram language model provides an XML Web Service to users to get the probability of a word sequence.
Although the n-gram corpus and the n-gram language model are easy to be applied in an application, those resources lack of other linguistic information such as parts of speech. Therefore, researchers  extended an English word n-gram corpus by adding parts of speech information and developed tools to accelerate the query speed. In this paper, we aim to develop a web-scale Chinese word N-gram corpus with parts of speech information, called NTU PN-Gram corpus, along the similar research direction. In contrast to English, Chinese web texts are harder to process. Besides the wellknown segmentation problem in Chinese, character encoding, the difference between simplified and traditional Chinese, mixing of multiple languages such as Chinese and English, and the Chinese punctuations are issues that need to be dealt with in the development of a web-scale Chinese PN-Gram (N-Grams with POS) corpus.
This paper is organized as follows. We describe the ClueWeb09 dataset in Section 2 with some statistics. In Section 3, we present an approach to resolve the issues of character encoding and mixing of multiple languages. In Section 4, we specify the segmentation tools and parts of speech tagging tool we used. In Section 5, the strategies to extract the word N-grams are illustrated. In Section 6, some statistics about the resulting dataset are shown.

ClueWeb09 Dataset
In 2009 CMU created ClueWeb09 dataset 1 to support information retrieval and natural language processing researches. They crawled 1,040,809,705 web pages in 10 languages. Of these, there are 177,489,357 Chinese pages, i.e., around 17.05% pages are in Chinese. The English data are encoded in UTF-8 format, but the data of the other languages are encoded in different encodings which depend on different situations. Those pages are stored in gzipped WARC format and are easy to read by using programming languages such as Java.
This dataset is huge -it has 25 TB (uncompressed) and 5 TB (compressed). Therefore, it is a good dataset for us to get the raw Chinese web pages. In this scale, processing the dataset may beyond the reach of many academic laboratories. For example, if it takes 1 second to segment and tag POS for a Chinese web page, it will take 5.6 years by a computer with a single node. In this study, we adopt a computer cluster to increase the processing speed.

Encodings and Mixing Languages
In Chinese, web developers use many charsets and encodings to represent their web pages. For example, in traditional Chinese, there are charsets such as Big5, CNS 11643, and Unicode. In simplified Chinese, there are charsets such as GBK, GB2312, and Unicode. Many charsets also support different encoding schemes. For example, Unicode uses UTF-8, UTF-16 and others to fit different situations, e.g., space, machine, or processing considerations. Besides, it is common to mix different languages in a Chinese text. For example, software programmers always mix Chinese and English technological terms together. An English term can be a noun or a verb. Game players may mix Japanese terms in Chinese documents in a game forum of Nintendo. Those issues complicated the encoding detection and language identification.
We use an algorithm to deal with encoding detection and language identification. Given a web page, we explore the possible encodings in a specific order, convert the page from the guessed encoding into Unicode, and detect if the converted text is a valid Chinese text. The first encoding passing the test is regarded as the encoding of the web page. The details are described in Algorithm 1.

Algorithm 1. Encoding detection and language identification
Input: A web page Output: Its encoding scheme and the language it belongs 1. for each encoding E 2.
Convert the page from E to Unicode (UTF-16)

3.
Compute the percentage of U+FFFDs in the converted page

4.
if the page is a valid Chinese page then return the encoding and the related language At step 1, we try a list of encodings in a specific order. The order is determined by the web page itself and a global list of encodings. Li & Momoi (2001) proposed a Mozilla Character Detector (Chardet) based on the content of a web page. They tested their approach in 100 popular web sites, and resulted in 100% detection accuracy. Although we found the detection of Chardet is not perfect in large scale document set like ClueWeb09, it is still a good tool to start. We first adopt their toolkit to detect the possible encoding. If the encoding fails at step 4, then we explore HTTP header encoding and encoding information in HTML metadata of the given web page. If these explorations fail at step 4 again, we try a predefined list of encodings. The order of encodings is determined by the encoding frequency of web pages detected by Mozilla Chardet. Table 1 lists the first eight of the distribution of encodings in a reference corpus. Of course, the statistics just gives a reference because Mozilla Chardet may make wrong detection. From this Table,  At step 2, we adopt Java to convert the given page from the guessed encoding E to UTF-16. During conversion, the system uses U+FFFD to replace those characters that cannot have valid mappings. The 'Replacement' character U+FFFD is useful at step 3 to compute the ratio of (U+FFFD)s to the total tokens in the converted page. Note that Roman alphabets are much easier to be transformed, thus they are not counted in the total tokens. At step 4, we use a threshold to determine if the conversion is successful.
Simplified Chinese, Traditional Chinese, Japanese, and Korean (CJK) characters have their own code points in Unicode, which are grouped in consecutive range. We count the ratio of the characters in the given web page to determine its major language at step 4 and to filter out the non-Chinese documents. Finally, the total number of valid Chinese pages used in this study is 173,741,587.

Segmentation and POS Tagging
After a valid Chinese web page is determined, we use Jericho HTML Parser 2 to extract the pure text in RFC3676 format from this page. Next we use Stanford Chinese Word Segmenter (Tseng et al., 2005) and Stanford Log-linear Part-Of-Speech Tagger (Toutanova et al., 2003) to process the text. The tagger has been demonstrated to have the accuracy 94.13% on a combination of simplified Chinese and Hong Kong texts and 78.92% on unknown words. The POS tag set adopted is LDC Chinese Treebank POS tag set. For those traditional Chinese texts, we translate them into simplified Chinese by using character-based translation. The segmentation and the tagging are the most time consuming steps in the development procedure.

Extracting N-grams with POS
Not all sentences in a web page are useful for N-gram extraction. We propose two strategies to select sentences. First, a sentence is identified by using a full stop, an exclamation mark, or a question mark in full-width, half-width, or ideo-graph character forms. Next, we ignore those sentences which contain less than 3 words or 5 characters. Such short sentences are usually menu items or button names in a web page. A sentence start marker <S> and a sentence end marker </S> are further inserted into the beginning and the ending positions, respectively.
We transform some types of terms in a sentence into specific class names. Those numbers that are smaller than 11 digits are replaced with zeros. The English words are transformed to foreign words <FW>. The words containing only digits and alphabets are replaced with a class <CHSQ>. We consider those words that occur more than count 1 times as unigram tokens. The remaining words are converted into a class <UNK>.
All n-gram patterns (n in {2, 3, 4, 5}) that occur more than count n times are extracted and placed into the final corpus. We adopt the representation scheme similar to . The following shows an example.
有效 5991459 JJ | 10377337 VA | 4295950 AD | … It indicates that term 有效 (effective) occurs 5,991,459 times as JJ (other noun-modifier) and 10,377,337 times as VA (predicative adjective), and so on. Now we have to determine the occurrence threshold, count n (n in {1, 2, 3, 4, 5}), for each n-gram. The minimum occurrence count of unigrams in Google Chinese Web N-gram corpus is 200. In this paper, we adopt the same threshold, i.e., set count 1 to 200. It means a word will be regarded as a unigram token, when it occurs more than 200 times.
For bigram or higher, the threshold is comparatively harder to determine. If the occurrences of each POS sequence are 40 on average, and a bigram has 10 different POS tags, then the count 2 must be set to 400. That will result in much fewer n-gram patterns. On the other hand, if count 2 is set to 40, then the occurrence count of each POS is 4 on average when the bigram has 10 POS tags. This is not a reliable count to estimate the occurrences of POS tags. Considering the issue that the possible POS tags may vary largely across different bi-grams, it is hard to ensure that each POS has a minimum occurrence. Therefore, we leave the issue of finding reliable count of POS to corpus users, and consider the occurrence counts of n-grams only. We believe that corpus users will make the best choice to fit their applications.
The accumulated number of bi-grams is shown in Figure 1. If the minimum frequency is 1, it will contain all bi-grams. We set count 2 to 40 (the same as Google N-grams) to get a reliable n-gram count. Other count n are also set to 40 for the same reason.  Table 2 further shows the statistics of the word N-grams. The numbers of unique entries are listed in the 2 nd column for each N. The number of unique entries whose frequencies are equal to 1 is listed in the 3 rd column. The last column shows the number of unique entries of frequencies larger than 2,000.  ,184,637,707 1,873,354,994 5,734,370 4 10,923,797,159 5,505,549,988 4,724,136 5 17,098,062,929 9,381,561,487 3,274,034 Table 2: Statistics of the word N-grams

Statistics of NTU PN-Gram Corpus
The statistics of the resulting dataset is shown in Table 3. Total 48.59% of Google Chinese unigrams can be found in our unigrams. The number of unigrams we extracted is much larger than that of Google dataset. One possible reason is that the segmentation tools adopted in these two datasets are different. Google Chinese Web N-gram corpus uses a fixed vocabulary set with frequency and finds the highest frequency product from all possible segmentations. In contrast, we use CRF classifier to segment sentences. The length distribution of the words in NTU PN-Gram corpus and Google Chinese Web N-gram corpus is specified in Table 5 The above examples suggest that we need a text filtering approach similar to Gao et al. (2002) to get a more clean result. Furthermore, a named entity recognition module is preferred to improve the segmentation performance. Nevertheless, this kind of data is valuable for researchers to study the segmentation problems in handling the texts on the web.

Conclusion
We develop a Chinese word N-gram corpus with parts of speech information in this paper. Some important issues in preparing such a corpus are addressed and discussed.
In the future, we will provide a tool similar to  to speed up searching the PN-Gram corpus. Besides, we plan to apply this corpus to NLP applications such as sentiment orientation detection. We hope that this public available corpus can boost the performance of NLP applications.