{'fraction_non_alphanumeric': 0.03311867525298988, 'fraction_numerical': 0.015026065624041705, 'mean_word_length': 4.603951890034364, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A new dictionary structure for bi-directional MT system

The importance and structure of MT-dictionary were discussed extensively by many researchers in machine, translation in the past. These structures were mainly concerned with MT-dictionaries for one-way translation systems. In the present paper, a new dictionary structure for bi-directional machine translation is being introduced. The new structure is being tested for Chinese-English as well as English-Chinese machine translation.


INTRODUCTION
The importance and structure of MT-dictionary were discussed extensively by many researchers in machine translation in the past (Knowles 1982, Lamb and Jacobsen 1966, Liu 1982, Loh 1975, Oettinger 1960, Wang 1982, Wang, T'sou and Chan 1971. These structures were mainly concerned with MT-dictionaries for one-way translation systems. Dictionary structures for bi-directional or multi-language machine translation systems were rarely discussed. The aim of this paper is to introduce a dictionary structure suitable for multi-language translation system. The dictionary structure was designed in conjunction with the Dual Language Translator (DLT) developed at the Chinese University of Hong Kong in 1978 (Loh, Hung and Kong 1978).

BASIC STRUCTURE
It is generally agreed that for translation from one language L1 into another language L2, the MT-dictionary D 12 must contain the following informations: is a set of internal codings of the source lexical items in L1, GI L1 is a set of grammatical informations of these items, IC L2 is a set of target equivalences (the target lexical items in L2) of these items, GI L2 is a set of grammatical informations of these target equivalences.
Similarly, for translation from language L2 into language L1, the MT-dictionary D 21 must also contain these types of informations.
For a one-way translation system, this kind of dictionary structure may seem to be quite suitable. However, for a bi-directional language translation system, this kind of dictionary structure requires almost identical information to be kept in two different storages which is redundant and undesirable. A new structure for the dictionary is thus required.
In the course of designing the Dual Language Translator (DLT) we had foreseen this problem and realized that only two types of information are stored in D 12 or D 21 , namely, the source language information and the target language information. Thus these two dictionaries D 12 and D 21 for the bi-directional translation system between the languages L1 and L2 may then be replaced by the dictionaries D l = { IC L1 , GI L1 } and D 2 = { IC L2 , GI L2 } together with some relations between D 1 and D 2 .
Bearing these points in mind, we proposed a structure for the MT-dictionary of a multi-language translation system.
The basic organization of the proposed MT-dictionary is illustrated in Fig. 2.1. The two main components of the dictionary are the DICTIONARY ADMINSTRATOR and the n SUB-DICTIONARIES.  The reason for separating these items from the others is to speed up the translation process. During the translation process, these items will be kept in computer main memory.
Thus dictionary consultation for these items will not be necessary, consequently reduce the times for lexical analysis. Due to the limitation of the size of computer main memory, the number of these special items is limited.
The information attached to each item, either special or regular, may be grouped into three different types of records: (a). Lexical information records,

(b). Grammatical and target information records, and
(c). Associated information records.
These three types of records are linked together internally by using pointers.
Lexical information records are used for lexical analysis.
A traditional method for representing lexical information is by means of a linear list such as illustrated in Fig. 3.2.
The disadvantages of this method are that duplication of lexical information exists and the search for a lexical item may have to be carried out linearly. We can rewrite the same list in Fig. 3.2 into a tree (Fig. 3.3), and a linked list representation of such a tree is given in Fig. 3.4. It is by linked list that the lexical information in the SUB-DICTIONARIES are represented. The format of the lexical information records is illustrated in Fig. 3.5. The grammatical and target information records are used for syntactical/semantical analysis and target determination.
The format of this type of records is illustrated in Fig. 3.6.

Fig. 3.6 Grammatical and target information records
The associated information records are used for determining the particular properties of the items, such as special article or measure word required, etc. and their format is illustrated in Fig. 3.7. (1). This item can be noun or a verb.
(2). If it is a noun then it has one meaning, and can be assigned the semantic category NA (non-animate).
For this particular meaning, the item has an associated information which specifies that it requires the particular measure word " (7309)".
(3). If it is a verb then it has one meaning, and can be assigned the semantic category HA (humanized action).
For this particular meaning, the item does not have an associated information.
According to the above specification, the lexical information record, grammatical and target information record and the associated information record of the item will be as shown in Fig. 4.1(a).
The COMMON DATA-POOL of the Chinese SUB-DICTIONARY may be a set of Chinese characters which might be necessary to be inserted into the Chinese sentence.
Consider the English lexical item "STUDY".
(1). This item can be a noun or a verb.
(2). If it is a noun then it has one meaning, and can be assigned the semantic category NA (non-animate).
For this particular meaning, the item has an associated information which specifies that the plural form of the item is "STUD + IES".
(3). If it is a verb then it has one meaning, and can be assigned the semantic category HA (humanized action).
For this particular meaning, the item has a, set of associated information which specify how the various form of the item can be constructed.
According to the above specification, the lexical information record, grammatical and target information record and the associated information record of the item will be as shown in Fig 4.1(b).
The COMMON DATA-POOL of the English SUB-DICTIONARY may be a set of characters which might be necessary to be inserted into the English words or sentence.