{'fraction_non_alphanumeric': 0.030670892195305367, 'fraction_numerical': 0.00827200945372509, 'mean_word_length': 4.7569573283859, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The comparison of Wiktionary thesauri transformed into the machine-readable format

Wiktionary is a unique, peculiar, valuable and original resource for natural language processing (NLP). The paper describes an open-source Wiktionary parser: its architecture and requirements followed by a description of Wiktionary features to be taken into account, some open problems of Wiktionary and the parser. The current implementation of the parser extracts the definitions, semantic relations, and translations from English and Russian Wiktionaries. The paper's goal is to interest researchers (1) in using the constructed machine-readable dictionary for different NLP tasks, (2) in extending the software to parse 170 still unused Wiktionaries. The comparison of a number and types of semantic relations, a number of definitions, and a number of translations in the English Wiktionary and the Russian Wiktionary has been carried out. It was found that the number of semantic relations in the English Wiktionary is larger by 1.57 times than in Russian (157 and 100 thousands). But the Russian Wiktionary has more"rich"entries (with a big number of semantic relations), e.g. the number of entries with three or more semantic relations is larger by 1.63 times than in the English Wiktionary. Upon comparison, it was found out the methodological shortcomings of the Wiktionary.


INTRODUCTION
The creation of machine-readable dictionaries is an important step in the road of the automatic text processing. Machine-readable dictionaries and Wikipedias, and Wiktionaries are heavily used in different disciplines, including ontology building (Wandmacher et al. 2007), machine translation (Etzioni et al. 2007; Muller and Gurevych 2008), automatic text simplification (Napoles and Dredze 2010), image search (Etzioni et al. 2007), and word sense disambiguation (Krovetz and Croft 1989).
Wiktionary is a unique resource and it could be useful for a wide variety of NLP tasks. But it cannot be used directly. There is a need to develop a software tool which makes it possible to convert the Wiktionary articles into a more suitable form for computer manipulation and processing, such as machinereadable dictionary (MRD).
This conversion software presented in the paper is a parser of the Wiktionary. The parser software is released under an open source license agreement (GPL), to facilitate its dissemination, modification and upgrades, to draw researchers and programmers into parsing other Wiktionaries, not only Russian and English. 2 The specificity of Wiktionary is that it is created by the community of enthusiasts, and it is probably that not all of them are professional lexicographers.
The structure of a dictionary entry is gradually, but constantly changing, since community experts regularly discuss and work out new and better rules. Also it should be taken into account that Wiktionary is permanently growing in number of entries and in the scope of languages. Now English Wiktionary contains about 740 different languages, and the parser recognizes 540 language codes.
Thus, the parsing of this linguistic resource makes high and specific demands on the software to be developed. These requirements are described in the next section.
The third section is concerned with the architecture of the Wiktionary parser. The comparisons of the main properties of Wiktionaries and their thesauri are presented in the fourth section. Discussion and related work conclude the paper.

REQUIREMENTS AND SOLUTIONS
Before the creation of the Wiktionary parser, the open-source software for the Wikipedia data extraction, the wikitext parsing, the indexing of Wikipedia texts (Krizhanovsky 2008), the search for related terms by analyzing Wikipedia internal links structure (Krizhanovsky 2006) was developed in our lab. This software can be used to create a parser, when the following conditions are met:  Software must be written in the Java programming language.
 Wiktionary dump should be passed to the MySQL database.
Requirement. Some requirements (problem statements) will be formulated for the successful design and implementation of a stable, functional, fast, modular and extensible parser.
Solution. The required solution with implementation remarks will be described.
The requirements to the parser software code, to the structure of the parsed Wiktionary database and to the development process are listed below.
Reliability and stability. At this stage of the Wiktionary development there are no special MediaWiki features in order to control and prevent the malicious or erroneous input data. For example, the user can type a language code which does not exist or user can enter an extremely long definition sentence. Therefore the parser should properly treat the errors and defects.
These requirements are satisfied due to the testing and the visualization (see below). Unit testing, as a method of extreme programming, is used, that is each non-trivial function is accompanied by one or more tests. The Flexibility. The Wiktionary formatting rules are constantly improving and changing. But if some rule is changed then the format and structure of all 1.5 millions of entries, even with the help of an army of bots, will not be changed at once and simultaneously in order to conform to the new rules. Only those entries which were edited after an adoption of a new rule will have an entirely new structure. A great number of entries corresponds to the preceding and outdated formatting rules, and this sad state of affairs can last for years until a volunteer editor will fix it. 3 Then the parser should be flexible enough to satisfy several formatting standards and yet to extract data from Wiktionary articles.
The testing helps again. The inputs of the unit tests are the parts of entries with different formatting standards.
Visualization. The visual examination of the data from the parsed Wiktionary database will prompt for the missing fields which have not been extracted by the parser. That is, there is a need in a tool to quickly glance at all the sections of a Wiktionary entry saved in the MRD or to see that there are some problems. The visualization will help to avoid the formulation of the tedious and low-level SQL queries.
The application wiwordik was being developed at the same time that the parser was being created. wiwordik is a visual interface (in the JavaFX programming language) to the machine-readable dictionary. It allows searching by the name of a Wiktionary entry or the word from the translation section. All the information extracted from the corresponding Wiktionary entry for the given word is presented to the user (Fig. 1).
Wiktionary ++ (breadth-first growing). It should not be difficult for the developer to add modules for the parsing of new Wiktionaries. This extensive growing of the parser (with a minimum of the additional and repeated work to be avoided) requires a clear and unambiguous division of the parser code into two parts: (1) the kernel, i.e. the part, which does not depend on the language, and (2) the language-dependent part, which have to be written anew for each added Wiktionary.
The current parser implementation already works with two Wiktionaries: Russian and English. The language (Russian or English) is one of the input parameters of the parser (Fig. 2, see -Input‖). Apparently, the parser has been already developed on the foundation of an incremental approach. That is, first of all, the text of a Wiktionary entry is split into high-level parts (Language, in accordance with WT:ELE rules 5 ), then the language sections are further split into Etymology subsections, then -Parts of speech, and so on. Thus, if there is a need, for example, to extract data from the Pronunciation section, then it is known which part of the parser code should be extended.
Integration. There is an important task to integrate the data extracted from different Wiktionaries into a common database. This integration is needed since dictionaries are built by hand and every Wiktionary contains unique data which are absent in other Wiktionaries.
It is an open question. On the one hand, our parsed Wiktionary databases (i.e. the database of the machine-readable dictionary) have the same structure ( Fig. 3), so a practical question is how to merge the two databases into a single database. But, on the other hand, the different Wiktionaries may contain duplicate, contradictory, and inconsistent data, so the dictionary merging becomes an interesting theoretical question.
Speed. There is an important need in a rapid parsing of the Wiktionary dump in a reasonable time. Since the Wiktionary is constantly growing, the regular and frequent parsing of the dump is required in order to support the MRD in the actual state.

ARCHITECTURE
Three rectangles drawn in the figure (Fig. 2) denote the following: the data processed by the program (parser), the program's input parameters and the modules of the program. Just one of these dumps uploaded to the local MySQL database is a source material, which will be processed by the parser in order to get the desired result, i.e. the parsed Wiktionary database (Fig. 2). In this resulting database the dictionary data are strongly structured, that is every lexicographic entity (definition, translation, etc.) has its own table in the database (Fig. 3).
Parameters. The meaning of the parameters is as follows (-Input", the top part of the Fig. 2): -the native language is the main language for the Wiktionary, e.g. Russian in the Russian Wiktionary; -the configuration parameters to access to both databases (the source and the receiver databases); -the number of the record in the source database, which give the possibility to stop and start again the parsing from this record.
The parser treats a Wiktionary article in three steps: (1) Extraction. The title and the text of an article from the Wiktionary are extracted. This functionality has been implemented in the Java package wikipedia.sql.
(2) Analysis. The text of an article is analyzed. Many times (during the pass) various regular expressions are used to extract the desired information from the text. This extraction and analysis are possible only due to the known structure of the article and due to the applying of the templates. 6 The more strict and rigid structure of the entry is adopted by a community of wiki editors, the more simple and reliable will the parser's algorithms be. The more number of templates widely used in the Wiktionary is, the more easy to extract the structured data from it.
The known project DBpedia (Bizer et al. 2009) also relies on the templates in the Wikipedia to extract the data from the encyclopedia.
During the analysis of an article, a temporary intermediate Java object (the class WordBase in a Java package wikt.word) is created. The hierarchy of subclasses of this class corresponds to the structure of the Wiktionary article (the source) and to the tables in the created database (the receiver).
The successful filling of this object with the extracted data is a prerequisite for the next step.

Comparison of the main properties of Wiktionaries
The main, from our point of view, properties of Wiktionaries and the sizes of the built machine-readable dictionaries are presented in the   (B) Machine-readable dictionaries filled by data from Wiktionaries, the size of tables (Fig. 3) The date of the Wiktionary dump Jan 6 2010 April 5 2010 - The But the tables which have small differences are also interesting, e.g. the table -relation‖ (the ratio of sizes is 1.57) and the table -translation‖ (the ratio is 1.55).
The following section will investigate the table -relation‖ in more detail.
In this paper the data extracted from the section -Translation‖ of the Wiktionary entry (and stored to the table -translation‖) are not analyzed, since there are a lot of language codes in the English Wiktionary to be added to the parser code (about several hundreds). Nevertheless, on the basis of these preliminary data, the numbers of translations extracted from the English Wiktionary 8 and from the Russian Wiktionary 9 are available online.

The comparison of Wiktionary semantic relations
The table -relation‖ of the machine-readable dictionary binds a type of a semantic relation (the table -relation_type‖), a certain meaning of the word (the table -meaning‖) and the wikitext (the table -wiki_text‖) which consists of synonyms, antonyms, etc. (Fig. 2).
The size of the table -relation‖ is the number of semantic relations extracted from Wiktionary. 157 thousands of such relations were extracted from the English Wiktionary, 100 thousands from the Russian Wikipedia.
With the help of MRD the number of semantic relations per dictionary entry was counted. The result is presented in the (Fig. 4). E.g. the English noun -toe‖ 10 contains 7 semantic relations spread across 6 types of semantic relations (synonyms, antonyms, hyponyms, holonyms, meronyms, coordinate terms).
It should be noted that the number of semantic relations in the Fig. 4 is calculated separately for each homonym in spite of the fact that homonyms are included in one Wiktionary entry. E.g. the first English noun in the entry -paw‖ 11 has 12 words within 4 types of semantic relations; the second noun has 7 words in 5 types of semantic relations.

DISCUSSION
In this section the features of the Wiktionary as a linguistic resource and the Wiktionary characteristics essential for the parser are under consideration. Also the current state of the parser is discussed.

The problems and shortcomings of the parser
The difficulty of the parsing is that the goal of the Wiktionary is to create a multilingual free content dictionary. At this moment the parser recognizes only 479 pairs (language code -language name) out of 974 languages of the English The parser omits entries which are written in unknown languages. So the number of words, meanings and translations presented in the first table will increase after adding the missing language codes.

Number of meanings, word forms, lemmas, and sly bots
In some wiktionaries the word forms (e.g. -dog‖ and -dogs‖) are described as different entries. The English Wiktionary entries describing these word forms (e.g. -selects‖, -militias‖) are created usually by bots, i.e. in an automatic manner.
Of course, it is a good and convenient addition to the dictionary, but… While counting the number of all entries, such word form was taken as the full value entry. Perhaps, this is the reason of the big discrepancies between the ratios of parameters of two wiktionaries in Table 1, since in the Russian Wiktionary the rule is not to create a separate entry for each word form (all the more automatically), only for lemma.
Thus, one of the high priority tasks is to -teach‖ the parser to distinguish between the usual entry and the word form entry, which points to the entry with lemma (a kind of -soft redirect‖). This distinction ability will allow comparing the real size of dictionaries.

Why the English Wiktionary is beaten by the Russian Wiktionary by a number of semantic relations?
The many parameters of the English Wiktionary outnumbered the parameters of the Russian Wiktionary by more than three to one ratio (3-8 times, see Table 1).
However, the number of semantic relations in the English Wiktionary is larger only by a factor of 1.57. Moreover, the percentage ratio of a number of entries with semantic relations to a number of content pages is almost twice as large for the Russian Wiktionary compared to the English Wiktionary.
We can propose the following hypothesis in order to explain these

Explicit language names as a source of errors
From the parser developer's point of view, the manual writing explicitly the language names (in order to define language of the entry, or in the translation section) is an unqualified and undoubted evil, since it gives room to make an error (misprint, misspelling). However, the editor will not be informed about this error.
Let us consider the fragments of the entries -bush‖ 17 and «ангел» 18 (Table 3). In the English Wiktionary the language of the word in question is written in the explicit form, in the Russian Wiktionary the special templates are used (rows 1-2, Table 3).  Thus, from the parser development point of view, the Russian edition of the Wiktionary is more professional in this field, since (i) an entry language header and (ii) translation section use the rigid structure of templates. It requires more effort from the user (the editor needs to keep in mind the language codes), but it's more fail-safe. So, if the user enters -et‖ instead of -es‖, then immediately he will see the language name -Spanish‖ instead of -Estonian‖. He will see and correct the misprint.

Data of each Wiktionary is unique
It was mentioned more than once previously that each Wiktionary contains unique data, which is absent from other Wiktionaries.
This statement may be supported by the following fact. There are only 3.9 thousands of semantic relations between Russian words in the English Wiktionary, but there are 84 thousands in the Russian Wiktionary. And vice versa, there are 3.4 thousands of semantic relations between English words in the Russian Wiktionary, but there are 44 thousands in the English Wiktionary.
The following tasks will be left as exercises for the reader (or for the author) to support or refute this hypothesis by comparing the built machinereadable dictionaries: -Find the degree of covering / intersection of words which belong to different languages, words with meanings, and the presence of semantic relations.
-Create a list of languages which are unique or almost unique, i.e. they are present only in one dictionary (Red List of Languages).
-Create two ordered lists: a list of languages which are better presented in one Wiktionary by different parameters (number of meanings, semantic relations), and a list for another Wiktionary.

RELATED WORK
Undoubtedly, the task of creating of machine-readable dictionaries existed long before Wiktionaries come on the scene (Krovetz and Croft 1989;Wilms 1990

Conclusion
The architecture of the extensible and modular Wiktionary parser was developed.
The modules for the extraction of three types of data from the Wiktionary entries There are many attractive ways to develop the parser and applications based on it.
But, in the first place, the Graphical User Interface for the created machinereadable dictionary based on data from the English Wiktionary should be developed. This interface for the Russian Wiktionary is ready and available online. 20