{'fraction_non_alphanumeric': 0.025399884370784352, 'fraction_numerical': 0.007901329735979958, 'mean_word_length': 4.773475745438362, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Dependency Network Syntax: From Dependency Treebanks to a Classification of Chinese Function Words

This article presents a new approach of using dependency treebanks in theoretical syntactic research: the view of dependency treebanks as combined networks. This allows the usage of advanced tools for network analysis that quite easily provide novel insight into the syntactic structure of language. As an example of this approach, we will show how the network approach can provide clear structural distinctions among the Chinese function words, which are very difficult to obtain directly from the original treebank. We hope to illustrate the enormous potential of the language network approach through a simple example.


Why treebanks?
Treebanks are the latest hype in linguistics. The interest in treebanks can roughly be explained by two main charms: the NLP push to data driven approaches and the linguist's fascination of creating a treebank following specific theoretical principles.
In greater detail, we can observe that Natural Language Processing requires treebanks for all kinds of data-driven approaches ranging from Machine Translation to text classification. Great efforts, monetary and personal, go into the creation of treebanks or the transformation of existing treebanks into new formats. In particular dependency treebanks offer interesting connections between texts and the representation of meaning, the ultimate goal of Computational Linguistics. This NLP interest in dependency treebanks has also enthused (and frequently financed) the community of "pure" linguists, who have discovered that the creation of coherent treebanks is linguistically challenging and fascinating. Work has been done on error detection (Dickinson & Meurers 2003), alignment of multilingual (Lopez et al. 2002) and of multi-stratal treebanks (Böhmová et al. 2003;Mille & Wanner 2010), on written and spoken data, just to name a few. The creation of a treebank can also have a unifying effect on a linguistic community by providing a reference analysis, other analysis have to be compared to (Penn Tree Bank, Marcus et al. 1993). But the creation of a treebank following a specific syntactic theory cannot in itself be considered as a confirmation of this theory (other than being a sociological proof of the existence of sufficient support for the theory to be able to create a treebank).
What is crucially missing in this picture is the usage of treebanks for linguistic discovery and theory confirmation or refutation that goes beyond searching for examples in the annotated data. Simple concordancers exist, some of them with sophisticated query languages (Zeldes et al. 2009) but it is up to the syntactician to go through the results and make conclusions. No generally accepted approach on how to interpret this type of data has been established.
The community of "corpus linguistics" is nearly exclusively busy with statistical analysis on pure text corpora, using tools like Wordsmith or Lexico3. At most, they use POS tagged corpora, often simply to disambiguate word usages. This domain of research has achieved impressive results in historical linguistics, sociolinguistics, and other domains where large amounts of data finally are systematically ploughed through (Baker 1993;Charteris-Black 2004). However, the mentioned tools and methods can not easily be applied to treebanks because first, the structure of the data is very different, and secondly, the limited size of treebanks compared to the vast amounts of unannotated text, makes a statistical approach less interesting.
Notable exceptions to this rule include work on usually small hand-coded treebanks like the ones used in Liu et al. (2009) for the study of dependency distance and in Liu (2009a) for the research of probability distribution of dependencies, where the traditional statistical approaches have shown their potential in theoretic syntactic research. As an emerging statistical method, the network approach brings a new angle to this type of research.
In this paper, we attempt to illuminate the network view of dependency treebanks. We will show how this approach reduces the diffi-culties in exploiting treebank data and how this approach can be successfully applied to small dependency treebanks, reducing the size limitations of existing dependency treebanks.

Language networks
The basic idea underlying dependency networks is very simple: instead of viewing the trees as linearly aligned on the sentences of the corpus, we fuse together each occurrence of the same word to a unique node, thus creating a unique and (commonly) connected network of words, in which the tokens are the vertices and dependency relations are the edges or arcs. This connected network is then ready to undergo common network analysis with tools like UCINET , PAJEK (Nooy et al. 2005), NETDRAW (Borgatti 2002), CYTOSCAPE (Shannon 2003), and so on.
In reality, extracting a network from a dependency treebank is slightly more complicated, as we have to use some heuristics to fuse together only the words that belong to the same lexeme (same category, near meaning). We refer to Liu (2008) for a description of multiple ways of network creation from dependency treebanks.
Linguistic research with using modern network analysis tools is an upcoming domain. The first conference on this subject, Modeling Linguistic Networks, was held in December 2012 in Frankfurt and united nearly 40 scholars from 14 countries. This community is guided by two assumptions: First, Language is physiognomicly a network and modeling of language should follow this guiding principle, and secondly, computational tools that have proven to be successful in sociology and computer science can be used for language networks, too.
The key interest of the network approach in linguistic research is that it provides a new way to analyze language systems. A central assumption of modern linguistic theories is that language is a system (Kretzschmar 2009). This widely accepted point of view, however, has remained on a purely theoretic level due to the absence of an operational methodology, until corpora and modern network analysis tools appeared. As language is a system, we expect there to be rules that cannot be predicted directly on the basis of the units. So looking at some specific words (or the relationship be-tween them) may not be an efficient way for discovering the global features of a language system. Modeling language as a network provides an operational way for observing the macroscopic features of language system and the relationship between the units and the whole system. For example, it can be used for determining the function or status of some units, such as words, in the language system as a whole.
Some research has been done on the structure of syntactic dependency networks ( Works on Chinese include networks that use as nodes the Chinese characters (Li & Zhou 2007;Peng et al. 2008), words and phrases (Li et al. 2005), phoneme and syllables (Yu et al. 2011;Peng et al. 2008), syntactic structure (Liu 2008;Liu 2010;Chen & Liu 2011;, semantic structure (Liu 2009b).
In general, the language network research, including that on Chinese language network, is developing rapidly in recent years. But the language network research inevitably has some aspects that need to be improved in order to establish this new domain. It seems that most of the language networks studies put a heavy emphasis on common features of various networks, such as 'small world' (Watts & Strogatz 1998)  这是一个橘子 zhe-shi-yi-ge-ju-zi 'this is an orange' co-occurrence networks (Li & Zhou 2007;Peng et al. 2008;Liu & Sun 2007;Li et al. 2005), resulting in research that lacks a strong connection to existing linguistic theories and research. But as more and more linguists get involved in the study of language networks, this situation is gradually changing.

The Chinese Dependency Network for this study
For the present work, we used the following treebank of Chinese: The treebank has 37,024 tokens and is composed of 2 sections of different styles:  "新闻联播" xin-wen-lian-bo 'news feeds' (name of a famous Chinese TV news program), hereinafter referred to as XWLB, is a transcription of the program. The text is usually read and the style of the language is quite formal. The section contains 17,061 words.  " 实 话实说" shi-hua-shi-shuo 'straight talk' (name of a famous Chinese talk show), hereinafter referred to as SHSS, is of more colloquial language type, containing spontaneous speech appearing in interviews of people of various social backgrounds, ranging from farmers to successful businessmen, The section contains 19, 963 words. Both sections have been annotated manually as described by Liu (2006). Table 1 shows the file format of this Chinese dependency treebank, which is similar to the CoNLL dependency format, although a bit more redundant (double information on the governor's POS) to allow for easy exploitation of the data in a spreadsheet and converting to language networks. The data can be represented as a dependency graph as shown in Figure 1.
The POS and dependency annotation is done on the transcribed texts. As the treebank contains different styles, it allows for general conclusions about the language, in spite of the lim-ited size of the corpus. Another benefit of the double nature of the data is that we can do comparative work based on these 2 sections. With words as nodes, dependencies as arcs, and the frequency of the dependencies as the value of arcs, we can build a network. For example, the sample shown in Figure 1 can be converted to a network as shown in Figure 2 (excluding punctuation).
Following the same principle, our Chinese treebank can be presented as Figure 3, an image that gives a broad overview of the global structure of the treebank. The resulting network has the following prop-erties: it is fully connected and there are no isolated vertices, it is a 'small word' and has a 'scale-free' structure. As we mentioned before, there are not many language characteristics that we can deduce directly from this big picture. What we need to do is to looki into the structure of some specific words in this big network, which in our study has brought about some interesting findings. The first step is to decide on the words we wanted to look into: the function words.

Chinese Function Words
Chinese is an isolating language: syntactic structure relies primarily on function words and word order rather than on rich morphological information to encode functional relations between elements (Levy & Manning 2003). Function words are words that have little lexical meaning or have ambiguous meaning, but instead express grammatical relationships with other words within a sentence, or specify the attitude or mood of the speaker" (Klammer et al. 2000). In Chinese, function words include prepositions, conjunctions, and auxiliary and modal particles (Yu 1998). As in any language, function words distinguish themselves not only by their syntactic properties, but also simply by their frequency. The words we are interested in are among the most common Chinese words: 在 1 zai '(to be located) in or at', 了 le 'perfective aspect marker or modal particle intensifying the preceding clause'.
We compared the frequent function words shown in XWLB, SHSS, and the Modern Chinese Frequency Dictionary and found that there are 3 function words that appear in all these 3 resources. They are: '的' de 'ablative cause suffix or possessive particle similar to the English genitive marker 's', zai and le. The frequency information of these 3 function words is shown in Table 2 2 . We will exclude de from this study because of its unique behavior 3 . We only chose zai and le as our research objects.
The differences in distribution between the two genres of texts are mostly based on the lexical poverty of spontaneous speech (SHSS) compared to written style, resulting in higher frequencies (of the smaller number of types) in the former genre. Moreover, the notably higher relative frequency of le in SHSS can be explained by the fact that one usage of le is an intensifier typical for the genre of spontaneous oral language. Inversely, zai can be omitted before locatives in oral Chinese. er with the preceding unit becoming an attribute or an expression referring to something or someone. Considering the complicated situations of the de-structure, they would require a special and extensive discussion, and are left for future research. 4 In Chinese, le and zai also can be content words even though these phenomena are not common. The Modern Chinese Frequency Dictionary doesn't distinguish these difference but we believe the deviation of the data won't change the fact that these 2 function words are among the most common Chinese words. This paper focuses on the structural distribution of linguistic units (words, in this study), more specifically of the function words zai and le. There is similar research on Chinese with different concerns: Liu (2007) analyzed the distribution of dependency relations and dependency distance in a Chinese treebank, including but not centered on function words. Gao (2010) and Gao et al. (2010) described the syntactic functions of nouns and verbs in mandarin Chinese, dividing syntactic functions of nouns and verbs into typical ones and atypical ones for a quantitative analysis.  tried to build a model of valency pattern from syntactic networks based on treebanks. All these works are done from the perspective of parts of speech instead of specific words. At the same time, there are several studies in Chinese concerned with specific words. For example,  have engaged in a study on the evolution process of the syntactic valency of the verb. They constructed three corpora of ancient classical Chinese, ancient vernacular and the modern vernacular, and selected ten verbs as the objects of their study to ascertain the diachronic behavior of these words. Even though they analyzed the complements and modifications of the words, they failed to give specific information about the complements and modifications, only distinguishing single word units from more complicated linguistic units. In contrast, our study provided more information of the words that can connect with zai and le. We analyzed the distribution of the dependents and governors of zai and le in XWLB and SHSS. The results are shown in Table 3 and Table 4.
Note that the genre differences are visible for both words: The governors of zai are very similarly diversely distributed for both genres but in spontaneous speech, the dependents of zai are much more diverse than in written style. Compared to zai, le has simpler combinatory possibilities (and no dependents), and here, the governors are more diverse in spontaneous speech than in written style.
Comparing these two words, we can see that, in general, zai can relate to more types of part of speech than le. However, it is not easy to interpret these tables and we will see that when passing to a network representation, the differences become much more easily accessible.  Dep-dependent, Govgovernor 6 Network properties of Chinese function words

Properties of '在' zai and '了' le
With the XWLB and SHSS syntactic networks, we studied the most frequently used network parameter of the words, the degree: The degree of a vertex (a word) refers to the number of its neighbors. This variable actually describes the number of different word types which are connected with a specific word. The directions of the arcs distinguish between indegree and outdegree. The indegree of a word is the number of arcs it receives while the outdegree is the number of arcs it sends. Reformulated linguistically, the indegree reflects the number of governors of a word and the outdegree, the number of the word's dependents. In our network, these two function words have the following properties in Table 5. Although the size of the original sections of XWLB and SHSS in the treebank is similar (in tokens), the size of the XWLB and SHSS net- Figure 4. The sub-network of '在' zai and its neighbors in the XWLB network works is quite different due to the difference in the lexical richness. In order to make the data more comparable, we standardized the original data, also shown in Table 5. The table clearly shows that: le has a zero outdegree because it cannot govern other words in our analysis of Chinese while zai has both indegree and outdegree; Besides, le's degree is higher in SHSS than XWLB which states that the combinatory possibilities of le is more diverse in spontaneous speech. On the contrary the distribution of word types that zai can connect with is more diverse in written style, especially obvious when it comes to the indegree.

Network Manipulation
To see the role that these 2 words play in the whole language network system, we carry out the following manipulations on the network: Since we are only concerned with the vertices connected to these two words, we removed all the vertices and arcs that are not connected to them. Figure 4 illustrates the graph of the remaining vertices and arcs of zai in XWLB.
Actually, we tried to do the same thing based on the original treebank. No doubt, the idea is workable but it is difficult to visualize the result. Since the words, which are either the governors or dependents of these function words, are numerous, it would take a very big table, more than 200 lines, to show all the detailed information. A more reasonable way to visualize the data, making it more readable, is making a graphical representation of the information, such as a scatter diagram or a network diagram as the one in Figure 4. In this diagram, we managed to arrange the words by the value of their arcs connected with zai. The words between circle ○ A and ○ B labeled with smallest vertices, far away from the center vertex zai, only connected with zai once in the treebank. The lines between these words and zai are numbered by the frequency of the connection shown in the treebank. Following the same principle, the words between circle ○ B and ○ C connected with zai twice in the treebank, and so they are nearer to the center vertex. The words between circle ○ C and ○ D connected zai three times and the words in circle ○ D , except the word zai itself, connected with zai more than three times in the treebank. The more connections there are between the words and zai, the bigger the size of the vertices representing the words, the shorter the distance between the words and zai. In this way, the diagram 4 clearly shows that, even though zai has many neighbors, most of them seem to prefer visiting it just once or twice, in other words, the number of connection is distributed more evenly among its neighbors. After removing the vertices, we combined all the words with the same part of speech except the two words we are studying. So we got a new "mixed language network". It mixes two types of vertices, one representing a word while the other one representing the part of speech. This new graph, as shown in Figure 5, also included the information of Table 3 and  Table 4. The results we got from analyzing the treebank can also be extracted from the language networks.
In this diagram, we put the arcs in different grayscales. The higher the value of an arc is, i.e. the frequency in Table 3 and Table 4, the darker their color, the bigger size of the vertices which represent the parts of speech. It is even easier to get the same conclusion than that drawn from Table 3 and Table 4: le can only be a dependent while zai can be a governor and a dependent; zai can relate to more types of part of speech than the le, zai can related to more words in XWLB than in SHSS while le can related to more words in SHSS than in XWLB. Now, we can see that the main difference between analysing the original treebank and the language network is that the language network can provide an easier and direct access to a graphic output, especially when the data is too complex or too big to be included in a limited table.
So are there other facts about the language structure that are extremely difficult to see di-rectly from the original treebank and that can be seen directly in the network? One of the advantages of the language network model is that it views the language as a connected whole system. Without the language network approach, describing the language system is more like talking about an unspecified abstract structure. The language network model gives a more specific structure model to the language system and also provides different computational tools that have proven to be successful in sociology and computer science, which are able to describe the different elements of a network system, or, as in our case, a language system. So we tried to manipulate the XWLB and SHSS networks to find out the roles of these two function words in the language networks systems. The way we tried actually follows a very simple logic. If you want to know the function of one element in a system, the simplest way is to remove it from the system and then to see what the consequences are: We respectively removed the vertices representing zai and le from XWLB and SHSS language networks and compared several most common features of the networks, the number of vertices, average degree, the number of isolated vertices, before and after removing the vertex.
The numbers of vertices are actually the numbers of word types in the treebank. Although the sizes of XWLB and SHSS are similar, the numbers of vertices of XWLB and SHSS networks, or the size of the networks, are obviously different due to the difference of lexical richness. The isolated vertices represent the vertices without any neighbors. This is the interesting part here. According to the data, there are no isolated vertices after removing le. All the remained vertices are still fully connected. So, if we believe the network somehow can be seen as the model of the syntactic structure of the language system drawn from this part of the treebank, then removing le seems to cause no significant trouble here. The whole structure didn't suffer from a systematic crisis, even though the le was a high frequency word with very high degrees. At the same time, removing zai caused isolated vertices in both XWLB and SHSS networks, especially in SHSS, even though the zai has lower frequency than le in the treebank and lower degrees in the network. In other words, removing this word created a much bigger systematic crisis. The reason is simple: le can only be a dependent. Take a picture like diagram 6: In the simple full connected network there is a vertex A that only has indegree and no outdegree. Because vertex A only attaches to other vertices and it doesn't convey any unique information between its neighbors, removing it from the network won't render any vertex isolated. This result fits a common sense in syntax that the governors are somehow more important than dependents when it comes to the structural completion of sentences. But it is very difficult to quantify the syntactic importance, especially for the whole treebank, text or language systems. We see that the analyzed function words, which share high frequency and degrees, in fact play very different roles in the system model: As a result, it seems safe to claim that zai is more important than le for this model's structure. The syntactic importance of specific words can be quantified in this way. Developing a numeric scale of a well-defined notion of "syntactic importance" is left for future research. This study shows that the language network approach can not only provide an easier and direct access to getting a graphic output but also can bring some fresh new angles for language analyzing.

Conclusion
This paper addresses the importance of developing techniques of treebank exploitation for syntactic research ranging from theorem verification to discovery of new relations invisible to the eye.
We advocate in particular the usage of network tools in this process and show how a treebank can, and, in our view, should be seen as a unique network.
We have shown in more detail, by opposing the function words zai and le, that the frequency of words is not equivalent to the word's importance in the syntactic structure, pointing to a notion that we may call the "centrality" of the word. The importance in the syntactic structure is still a vague notion that needs to be refined further, but simple network manipulations like removal of the words in question can reveal properties of the words that seem to be closely related to the words' structural roles. For example, a word A whose removal breaks the network in parts is clearly more important than a word B whose removal preserves the connectedness of the network (as the word only occupies exterior nodes). Since the results shown in this paper confirm well-known facts concerning these two function words, the same method can be applied to other function words as well content words. Ongoing research includes analyses of the Chinese equivalent of the following words: de 'ablative cause suffix or possessive particle similar to the English genitive marker 's', wo 'I, me, myself', shi 'are, am, yes', ge 'individual, entries', yi 'one, single', zhe 'this, it,these', bu 'do not, need not', ta 'he, him', shuo 'speak, talk, say', ren 'person, people, human being', and dao 'arrive, reach, get to'.
We leave it for further research to develop the notion of "centrality" into a numerical value that would allow comparing any pair of words. Equally, the active field of network analysis will in time reveal new techniques that have in turn to be applied to new and bigger language networks based on treebanks of different types and languages. This could establish network syntax as one branch of the emerging field of data-driven linguistics.