{'fraction_non_alphanumeric': 0.026433750823994726, 'fraction_numerical': 0.0064601186552406065, 'mean_word_length': 4.753128555176337, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 1}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Tools and resources for Tree Adjoining Grammars

This paper presents a workbench for Tree Adjoining Grammars that we are currently developing. This workbench includes several tools and resources based on the markup language XML, used as a convenient language to format and exchange linguistic resources.


Introduction
Our primary concern lies in the development of efficient parsers for various grammatical formalisms of interest for Natural Language Processing. Tree Adjoining Grammars [TAG] is one of these formalisms, important from a linguistic point of view but also because it is possible to design efficient parsers.
However, during our work on TAG, we were confronted with a lack of standardization of grammars, especially when dealing with wide coverage grammars. The XTAG System 1 (The XTAG Research Group, 1995) provides an implicit standard, but it is not very readable and lacks explicit specifications. The various grammars we studied presented many variations. Moreover, we also noted many problems of consistencies in most of them.
Following others, amongst whom LT XML 2 and especially (Bonhomme and Lopez, 2000), we considered that the markup language XML 3 would be a good choice to represent TAG, especially with the possibility of providing an explicit and logical specification via a DTD. Being textual, resources in XML can be read by humans and easily exchanged and maintained. Finally, there exists more and more supports to handle XML resources. We have also found that XML is a convenient language to store linguistic results, such as the shared derivation forests output by our TAG parsers.
The paper starts with a brief introduction to TAGs. Section 3 presents the different XML encodings that we have designed for the representation of grammars and derivation forests. Section 4 presents several different maintenance tools we are developing to handle grammars and derivation forests. Section 5 presents servers used to access different kind of informations. Interfaces for these servers are presented in Section 6.

Tree Adjoining Grammars
The TAG formalism (Joshi, 1987) is particularly suitable to describe many linguistic phenomena. A TAG is given by a set of elementary trees partitioned into initial trees and auxiliary trees. Internal nodes are labeled by non-terminals and leaves by non-terminal or terminals. Each auxiliary tree has a distinguished leaf, called its foot and labeled by a non-terminal, the same as the root node of .
Two operations may be used to derive trees from elementary trees. The first one, called substitution, replaces a leaf node labeled by a nonterminal ¡ by an initial tree ¢ whose root is also labeled by ¡ . The second operation, called ad-  Figure 1: Adjunction junction, is illustrated by Figure 1. An auxiliary tree whose root is labeled by ¡ may be adjoined at any node £ labeled by ¡ . The subtree rooted at £ is grafted to the foot of . Feature TAGs extend TAGs by attaching to nodes a pair of first-order terms represented by Feature Structures [FS] and called top and bottom arguments. These feature structures may be used, for instance, to handle agreement or enforce semantic restrictions.
Lexicalized (Feature) TAGs assumes that each elementary tree has at least one lexical node labeled by a terminal. However, explicit lexicalized grammars would be huge, with one or more elementary trees for each entry in the lexicon. The choice made by the XTAG group and by all the designers of wide coverage TAGs is to factorize the grammars and gives enough information to lexicalize parts of the grammars when needed. Morphological entries (or inflected forms) reference one or more lemma entries, which, in turn, refer to families of tree schema. A tree schema is an elementary tree with a distinguished leaf called anchor node that is to be replaced by a morphological entry. Each reference may be completed by additional constraints.
For instance, extracted from a small French grammar, Figure 2 shows the basic elements (morphological entry donne, lemma \DONNER\, and tree schema tn1pn2) used to build the tree tn1pn2(donne) corresponding to the syntactic pattern (1) and illustrated by sentence (2). The lemma part states that the subject NP¤ and the prepositional complement NP¥ must both be human and that NP¥ is introduced by the preposition à (co-anchoring). In the tree tn1pn2, the substitution nodes are marked with ¦ and the anchor node with § ©

Representing grammars
We have designed a DTD 4 that clearly specifies the relations between the various components of a grammar. For instance, the following DTD fragment states that a morphological entry is characterized by a field lex and includes zero or more description entries (for documentation) and at least one reference to a lemma (lemmaref). Similarly, an element lemmaref is characterized by the fields name and cat, and may be completed by a FS argument (fs).
Following the DTD, the various elements described in Figure 2 may be represented by the (tiny) following XML fragment, omitting the FS specification on nodes for sake of space and clarity.

Encoding derivations
A (deterministic) TAG parser may return either the result of the analysis as a parse tree, or the steps of the derivation as a derivation tree. These two alternatives are illustrated for sentence (3) by Figures 3 and 4 (with Figure 5 showing the elementary lexicalized trees). A derivation tree indicates which operation (substitution or adjunction of some tree) has taken place on which node of which tree, for instance the adjunction of tree a j oli at node labeled N. It is worth noting that the parse tree may be retrieved from the derivation tree, which motivates our interest in deriva-tion trees.  In case of ambiguity (frequent in NLP), several or even an unbounded number of derivation trees may actually be compacted into a shared derivation forest, equivalent to a Context-Free Grammar (Lang, 1991). This remark has guided the design This DTD introduces the primary elements op, deriv, and node as well as an element opref used to reference elements op. The logical organization of these elements is sketched in Figure 6. More precisely: op , identified by its attribute id, denotes either an operation of substitution or of adjunction (attribute type) on some syntactic category (attribute cat) for some span of the input string (attribute span). Sub-elements of op may also be present to specify the feature values associated to the operation.
deriv details a possible derivation for some operation, based on some lexicalized tree given by a tree schema (attribute tree) and an anchor (anchor).
node specifies which operation op has been performed on some node (attribute node_id) of an elementary tree during a derivation.
A derivation tree may be expressed in a nested way using only elements op, deriv, and node. A shared forest will require the use of opref to denote multiple occurrences of a same operation. 5 http://atoll.inria.fr/~clerger/forest.dtd,xml The above derivation tree may be represented by the following XML fragment (omitting information about the feature structures).

For the grammars
The XML encoding of grammars is convenient for maintenance and exchange. However, it does not correspond to the input formats expected by the two parser compilers we develop. One of them (DyALog) expects a prolog-like representation of the grammars (Alonso Pardo et al., 2000) while the second one expects Range Concatenation Grammars [RCG] (Boullier, 2000). Besides the conversion modules LP and RCG, we also have a read/write XML module. Module Checker is used to check (partially) the coherence of the grammar and to produce some statistics. Module Analyzer extracts information needed for the compilation by the DyALog system. Module Strip deletes all information relative to feature structures from the grammar. Module SQL may be used to store to and load from a SQL database.
Our choice of Perl has been motivated by the availability from archive sites of many Perl modules to handle XML resources or database access. Moreover, the development of a Perl module is fast (for a prototype), generally only a few hours. For instance, we have realized a prototype module LaTeX, useful to build the documentation of a grammar. We are also thinking of an HTML module to build online versions.

For the derivation forests
Similarly, we have also developed a set of modules to handle derivation forests (Fig. 8)  Other modules should be added in the future, such as SQL module to read to and to write from a database used as a derivation tree-bank, a Strip module to remove features, or different filtering modules to extract subsets of a forest.

Servers
Exploiting some of these modules, but also other components developed in Java, we are installing several servers to access different kinds of information (parsers, grammars, forests) in uniform and distributed ways.

A server of parsers
We are exploring several ways to build efficient parsers for TAGs (Éric Villemonte de la Clergerie and Alonso Pardo, 1998;Alonso Pardo et al., 2000;Éric Villemonte de la Clergerie, 2001;Boullier, 2000;Barthélemy et al., 2001;Barthélemy et al., 2000), which leads us to maintain a growing set of parsers. Moreover, we wish to be able to compare the output produced by these parsers to check soundness, completeness and level of sharing in the derivation forests. To achieve these objectives, we provide a uniform setting by installing a simple server of parsers, written in Perl. Once connected to this server, one selects a parser and sends a sentence to parse; the server returns the shared derivation forest in raw, HTML, XML, Tree or Dep formats.
A WEB front-end 7 may be used to connect to this server. Figures 9 and 10 show two views of a derivation forest built using the server.
Another WEB front-end 8 allows the direct submission to the server of a derivation forest in one of the 3 recognized input formats (LP, RCG, XTAG). Submission in XML format should be added soon.

A server of grammars
Because of the size of wide-coverage grammars, we believe that working with grammars stored in files is no longer a valid option (except for exchanging them). Databases should be used instead where bits of information can be efficiently retrieved. Moreover, modern database managers provide server fonctionalities and security mechanisms.
Around these ideas, we are currently developing in Java a server of grammars.
First the grammars are loaded into a SQL database (namely MySQL 9 ). It should be noted that the structure of the database reflects the XML structure for grammars and not directly the structure of the grammars. This means that the loading phase may be performed for (almost) any kind of XML documents.
The second main component of the server is a small query language used to fetch information from the database while hiding, for non specialists, the complexity of languages SQL, XQL or XPath. We have chosen an object oriented notation which, once again, reflects the structure of the TAG DTD and which is also close to path equations familiar to computational linguists in, for instance, HPSG. We have several types such as family, morph, tree or node corresponding to the different kinds of elements of the DTD. For each type, several methods are available. For example, the following query returns the name and the Database Id (DBId) of all trees belonging to family tn1pn2. A second kind of requests takes a DBId F and returns the full XML fragment associated to the XML element whose index in the database is The grammar server works as a Java servlet, integrated in a HTTP server Apache using JServ 10 . It may be accessed using URL with parameters (as done for CGI scripts). The server decodes the parameters and transforms the query into a SQL query send to the database corresponding to the selected grammar. The result is either a table encoded in XML format or an XML fragment of the grammar. A small WEB interface 11 is available to display the results in a navigator (by transforming them into HTML) but it is also possible to get the results as an XML file. Tools can therefore query the server by sending a URL and getting back the results in XML.
The server should be soon completed for edition tasks. Full deletion of an element and of its descendants may be achieved using DBIds. Addition can be achieved by sending an XML fragment and a DBId (stating where to attach the XM-L fragment).

A server of derivation forests
The development of a server of derivation forests has just started, along the same lines we followed for the server of grammars. Such a server will be an alternative to treebanks. Two main functionalities are planned for this server. First the possibility to add a derivation forest to the underlying database (in order to build corpora of derivations) and second, ways to access these derivations through a simple but powerful query language.

Interfaces
We have already mentioned WEB interfaces to the parser server and the grammar server. Besides these interfaces, we (Kaouane, 2000) have also modified and enriched a Java interface developed by Patrice Lopez (Lopez, 2000). The new version can import grammars and derivation forests that follow our XML DTD. It can also use the server of parsers, sending the sentence to parse and receiving the shared derivation forest in XML format.
The derivations are extracted from the derivation forests and, for each derivation, the tool displays both the derivation tree and the corresponding parse tree (see Figure 11). It is also possible to follow step by step a derivation by moving forward or backward. We found this functionality useful to understand or explain to students the workouts of TAG.
The viewer may also be used to browse the different components of a grammar (trees and lexicons), therefore helping in its maintenance.
This tool already exploits the parser server, but we also plan to extend it to exploit the grammar server (for browsing the grammars or displaying a derived tree) and the forest server (for accessing derivation forests).

Conclusion
The experiments we have performed have strengthened our opinion that XML is really adequate to maintain and exchange linguistic resources and that XML allows us to quickly develop tools to handle these resources. Most of the components presented in this paper have been developed over a short period of time, and, while still preliminary, are fully usable. We believe that XML and these tools gives us solid foundations to further develop a complete environment to handle TAGs, based on many simple and easyto-maintain tools (instead of having a monolithic system). We also think that the availability of such tools or resources may prove useful for linguists (to develop grammars with browsing and maintenance tools), for parser developers (to browse grammars and derivations), and for students in computational linguistics (to understand Tree Adjoining Grammars).
The tools and resources that we develop are freely available. Tools are based on a modular architecture with a specification given by the DTD and we hope that new components will be added by other people.