{'fraction_non_alphanumeric': 0.030343083730820068, 'fraction_numerical': 0.011953336015171541, 'mean_word_length': 5.091004550227511, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Training a Swedish Constituency Parser on Six Incompatible Treebanks

We investigate a transition-based parser that uses Eukalyptus, a function-tagged constituent treebank for Swedish which includes discontinuous constituents. In addition, we show that the accuracy of this parser can be improved by using a multitask learning architecture that makes it possible to train the parser on additional treebanks that use other annotation models.


Introduction
Syntactic parsing is a widely used intermediate step in several natural language processing (NLP) tasks. For the last couple of decades, syntactic parsing has largely been based on machine learning systems, trained in a supervised fashion using large collections of hand-annotated sentencestreebanks. While the accuracy of automatic parsers had largely reached a plateau a few years ago, the introduction of deep learning techniques has led to recent improvements in accuracy. Hand-annotated treebanks are based on linguistic models that can vary drastically. The most widely noted difference among annotation models for syntax is probably the divergence between constituency models that conceive of the sentence structure as consisting of hierarchically organized phrases or constituents, and dependency models that represent the sentence structure as a graph where the tokens (words) are the edges. To exemplify this divergence, Figure 1 shows the Swedish sentence Sånt tror jag inte på. 'I don't believe in that kind of stuff.' annotated according to constituency and dependency models. However, even within one class of models, there are many theoretical design choices that can vary between different treebanks. For syntactic parsing, as for supervised learning in general, the availability of a substantial number of annotated examples is crucial. Manual annotation is time-consuming and re-using previously annotated treebanks would therefore be beneficial. However, the variability of treebanks is a major nuisance: if we want to develop a parser that uses some particular annotation model, it is not evident that any other available treebanks can be used. This is an obvious problem for a language such as Swedish, for which the largest treebank (Nilsson et al., 2005) is significantly smaller than treebanks for e.g. English, Chinese, and German. At the same time multiple treebanks with different types of annotation -different types of encoded linguistic knowledge -are available. We are aware of at least five unique treebanks for Swedish, most of which are annotated according to their own respective models (some of them more than one model). Is there a remedy: can we develop an approach that can utilize additional treebanks? For non-neural, featurebased dependency parsers, Johansson (2013) proposed two  feature-based approaches to multi-treebank training and evaluated them using treebank pairs in four languages. The first approach was based on multitask learning (Caruana, 1997) using a shared feature representation (Daumé III, 2007), while the second approach used stacking or guided parsing (Nivre and McDonald, 2008). A more recent approach, based on treebank embeddings in a neural dependency parser, was presented by Stymne et al. (2018); however, their approach has so far only been evaluated for sets of treebanks that are very close in annotation style.
In this work, we present the first results for parsing with the Eukalyptus treebank of written Swedish (Adesam et al., 2018), a function-tagged constituency treebank including discontinuous constituents. Furthermore, we show how a transition-based neural parser can be improved by using a multitask architecture that allows us to train the parser using a number of auxiliary treebanks, some of which are dependency treebanks. We are not aware of any previous work that has utilized constituency treebanks in this context, and especially not in a training process that uses both constituency and dependency treebanks.

Neural Transition-based Parsing
The parsers considered in this work belong to the class of transition-based parsers. In this approach, the parser builds the output structure in a step-by-step fashion by executing actions in a state machine. This state machine uses a stack to store partially built structures and a buffer that keeps the remaining part of the input. Our parser uses a transition system based on the shift/promote/adjoin system for constituency parsing introduced by Cross and Huang (2016), to which Stanojević and Garrido Alhama (2017) added a swap transition (Nivre, 2009) to allow for discontinuous constituents (e.g. the PP in Figure 1a). This transition system allows five different actions: SHIFT, which moves an item from the buffer onto the stack; PRO-MOTE, which takes the top item of the stack and starts to build a constituent; LEFT-ADJOIN and RIGHT-ADJOIN, which attach an item to the left or to the right of a constituent, respectively; and SWAP, which moves the secondto-last item of the stack back into the buffer. Figure 2 states these actions formally; for brevity, we omit the RIGHT-ADJOIN action and the preconditions that determine whether an action is applicable. The model used in the parser by Stanojević and Garrido Alhama (2017) can be seen as a constituent-based variation of the model by Dyer et al. (2015). It relies on several variants of the long short-term memory (LSTM), a well-known model for representing sequential computations (Hochreiter and Schmidhuber, 1997). The components of the parsing model are the following: • word representations: a bidirectional LSTM applied to word and part-of-speech tag embeddings; 1 • constituent representations: complex linguistic units are represented by applying a tree LSTM (Tai et al., 2015) compositionally; • state representations: two separate stack LSTMs (Dyer et al., 2015) represent the stack and buffer, respectively; 2 • action selector: feedforward multiclass classifiers that determine the next action to execute, and the constituent label. Figure 3 shows the representation of an intermediate state when selecting an action during the generation of the tree in Figure 1a; the correct action in this situation would be to SWAP the token Sånt back into the buffer, in order to build the discontinuous constituent. The parser is trained using a static oracle, since no efficient dynamic oracle is known for this transition system (or its dependency counterpart, the arc-standard system). We note that dynamic oracles are available for closely related transition systems (Goldberg and Nivre, 2013), but require special care when a SWAP transition is used (de Lhoneux et al., 2017). We leave dynamic oracle training to future work. We extended the model by Stanojević and Garrido Alhama (2017) so that it outputs function edge labels as well, e.g. OO, HD, OA, etc. in Figure 1a. The edge labels are generated when executing the PROMOTE, LEFT-ADJOIN, and RIGHT-ADJOIN actions. Taking edge labels into account required modifications of the tree LSTM that builds constituent representations compositionally, as well as a new feedforward unit to predict the edge labels.

Multitask Learning for Neural Network Parsers
Johansson (2013) applied multitask learning, using a feature representation that is partly shared between tasks (Daumé III, 2007), to train a feature-based dependency parser using multiple incompatible treebanks. This approach is not directly applicable here, since the model does not use an explicit feature representation. Instead, we follow recent work in multitask learning for NLP (Ruder, 2017), and train parsers for the different treebanks where some components are shared between the tasks; the idea is that the shared parts of the models will represent the commonalities between the tasks, abstracting away from the low-level peculiarities.
In this work, we follow the simple intuition that any component that does not explicitly depend on a treebank annotation model is shared between the parsing models for the different treebanks. That is, the representation models for words (word embedding and bidirectional LSTM) and the buffer are shared, while the representations for constituents and the stack, as well as the action selector, are kept separate for each treebank. We leave a full exploration of the best way to select the shared components to future work.
This architecture in principle allows us to use different types of transition systems, for instance for processing constituency and dependency treebanks differently. In this work we simply treat dependency trees as a special case of constituent tree, using a "dummy" constituent label. We can then apply the same type of transition system and learning model when processing all treebanks.

The Eukalyptus Treebank of Swedish
Early treebanks for Swedish such as Talbanken (Einarsson, 1976) and Syntag (Järborg, 1986) were annotated with constituency structures, and the recently created treebank Eukalyptus (Adesam et al., 2018) also has a constituencybased syntactic description. This treebank contains around 100 000 tokens, distributed over approximately 5 500 sentences. These were chosen from five different contemporary Swedish text types, which are public domain. They range from formal to informal, and from informative to entertaining, including both news text and blogs.
Eukalyptus has been manually annotated with part-ofspeech tags, morphological features, word senses, and syntactic structure. The syntactic description is similar to for example the German NEGRA/TIGER scheme (Brants et al., 1999). Tokens are connected into phrases, and each child (edge) has a function label. Phrases may be discontinuous. Secondary edges are used to mark various types of shared information in constructions such as coordination and control. The syntactic description is described in Adesam et al. (2015a). The treebank also uses special phrase labels to connect multiword units, detailed in Adesam et al. (2015b). We distinguish two types of multiword units: analyzable -which have an internal syntactic representation, and where a multiword node is added to attach the multiword label, gathering the parts of the multiword unit with secondary edges -and unanalyzable -which do not receive a syntactic analysis but are attached to the tree through their multiword node. A full example tree can be seen in Figure 4. The parts of speech together with the phrases and syntactic functions create a whole, where the different levels of information have complementary roles. However, for the current paper we will focus on parts of the syntactic annotation.

Parsing with Different Types of Treebanks
As mentioned, there are several treebanks available for Swedish, annotated using several different annotation models. In this work, our primary goal is to build and improve a parser for the Eukalyptus treebank described in Section 4.. To achieve this we will experiment with adding different types of treebanks during training.

Treebanks Used in the Experiments
The Eukalyptus treebank is the primary treebank, which we use in the single-task experiment and that is used to compute the evaluation score. As auxiliary treebanks in multitask training, we use five different treebanks. Two of them are constituency-based: Talbanken05, which is a modernized conversion (Nilsson et al., 2005) of the original Talbanken (Einarsson, 1976); and Syntag (Järborg, 1986).
These two treebanks are annotated in a fairly flat style, in a manner similar to Eukalyptus and the TIGER treebank (Brants et al., 1999). The remaining treebanks are dependency treebanks, all part of the Universal Dependencies project (Nivre et al., 2016); we used all three available Swedish UD treebanks: Talbanken, LinES, and PUD. Table 1 shows the sizes and structural properties of all treebanks: whether they allow discontinuities and whether they include constituent or edge labels. Gold-standard part-of-speech annotations are available in all the treebanks, but use different annotation standards. Instead of using the gold-standard tags, we carried out the experiments using tags predicted automatically by Hun-Pos (Halácsy et al., 2007), using the tagset defined by the Stockholm-Umeå corpus (Ejerhed et al., 1992). In addition to being a more realistic setup, this had the added advantage that all treebanks could use the same set of part-of-speech tags. We leave multitask learning at the part-of-speech level to future work. In addition, a number of adaptations of the Eukalyptus treebank are carried out. First, secondary edges are ignored. In many cases this does not change the information available in the treebank. However, for shared information in for example coordinations, there are now parts missing. In addition, removing secondary edges affects the annotation of multiword units. For analyzed multiwords, only a unary node remains of the multiword annotation, which is therefore removed. Taking the secondary edges and analyzed multiword units into account is an interesting avenue for future research. Finally, punctuation, interjections, and discourse particles -items that were not part of the regular tree -are attached to make each sentence a complete tree. These units are attached as high as possible without introducing discontinuities.

Experimental Protocol
We evaluate the baseline (the parser described in §2.), which uses just the primary treebank, as well as three different multitask learning setups where auxiliary treebanks are added ( §3.): just dependency treebanks, just constituency treebanks, and all auxiliary treebanks. For each parser, we carry out a 10-fold cross-validation 3 over the primary treebank; for each fold, we run the parser twice, using different  random seeds, and select the model that gives the highest score on a development set. The experiment uses two different evaluation metrics: (1) the F-score for finding and labeling constituents, such as S (sentence) and PP (prepositional phrase) in Figure 1a; (2) the accuracy of labeling edges, such as SB (subject) and OO (object) in Figure 1a. When evaluating, predicted and a gold-standard constituents are considered equal if their labels match and their yields (sets of covered tokens) are equal. Following Blaheta and Charniak (2000), only the correctly parsed constituents are included when computing the edge labeling accuracy. We used the evaluation module of DISCODOP (van Cranenburgh et al., 2016) to compute all scores. Table 2 shows the results of the evaluation. The constituent F-score for the baseline parser, which was trained on the Eukalyptus treebank only, is about 6 absolute points lower than that reported for German by Stanojević and Garrido Alhama (2017). The lower score is unsurprising since this treebank is about 9 times smaller than the TIGER treebank used in their experiments. Moreover, the results show that all three multi-treebank parsers improve over the single-treebank baseline: they are significantly better at finding constituents. 4 The edge labeling accuracy, however, does not show any significant improvements. It seems that the main takeaway is "the more, the better," as the top-scoring setup uses all five auxiliary treebanks. However, we get a significantly stronger improvement from the constituency treebanks than from the dependency treebanks. This may partly be a size effect, because the combination of constituency treebanks is larger, but may also partly be because these treebanks provide a training signal sentences before dividing into cross-validation folds, each test fold will typically be sampled from a single genre, which will then also be under-represented in the training set. It seems likely that shuffling would give us slightly higher evaluation scores. 4 The p-values are less than 0.05 in all comparisons to the single-task baseline. We used an approximate randomization test. that is more relevant when learning to create and label constituents in the target treebank.

Conclusions
We trained a parser on Eukalyptus, a Swedish functiontagged constituency treebank including discontinuous constituents. The baseline parser is an extension of the implementation by Stanojević and Garrido Alhama (2017) that allows edge labels to be predicted. Of more general interest, we showed that this parser can be improved by adding auxiliary treebanks in a multitask learning setup. Even if we have the goal of predicting outputs that adhere to a specific annotation model, treebanks that are incompatible with our target model do not need to be wasted. Constituent treebanks seem to be more useful as auxiliary treebanks when training a constituent parser, although the dependency treebanks also give an improvement. This is our first investigation of multi-treebank training for neural transition-based constituency parsers and it remains an open research problem to fully explore the spectrum of sharing architectures and find the one that best utilizes the auxiliary treebanks. The importance of this design choice has been discussed extensively for other NLP tasks (Ruder, 2017); for instance, Søgaard and Goldberg (2016) designed a carefully crafted sharing architecture for sequence labeling tasks. Ruder et al. (2017) discuss a method to learn the sharing architecture. In addition, it would be useful to investigate how the utility of a multitask setup is affected by the size of the primary treebank (Johansson, 2013).