{'fraction_non_alphanumeric': 0.02716890771781175, 'fraction_numerical': 0.013814698839565298, 'mean_word_length': 4.989244346387204, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Generalizable Features Help Semantic Role Labeling

In this paper, we take on the challenge of developing effective generalizable features for the task of semantic role labeling in the constituency grammar framework. Based on the knowledge of argument structure, on the constraint imposed by context dependence defined in the theory of argument realization, and on the knowledge of moved and displaced core arguments, we design the base argument configuration (BAC) feature that generalizes across four types of syntactic structures involving moved and displaced core arguments. As part of the effort to derive this base argument configuration feature, we also identify the core and non-core arguments in the system which is the first case in the field of semantic role labeling. Together with two levels of backoff features, the BAC feature effectively solve the argument classification task. However, as the experimental results show, our overall performance is affected by the argument identification module at present.


Introduction
The task of semantic role labeling identifies the semantic argument(s) of a verb and assigns a semantic role label to each semantic argument. Since the seminal work on semantic role labeling (SRL) by Gildea and Jurafsky (2000), the community has seen much effort dedicated to the task, which generated dozens of published SRL systems, including the ones participated in the shared tasks on CoNLL-2004, CoNLL-2005, and CoNLL-2008 In the past eight years, although researchers approached the SRL problem from different perspectives, they all focused on determining the appropriate syntactic/semantic knowledge and machine learning system to tackle the challenges in SRL (Carreras and Marquez, 2005;Surdeanu et al., 2008).
In terms of searching for the proper syntactic/semantic knowledge, the SRL researchers explored features based on two formalisms, namely constituency grammar and dependency grammar. The SRL systems constructed between the years of 2000 and 2006 investigated a variety of features that constituency grammar provides for. The main stream in this line of work had been developing specific features to cover diverse syntactic configurations that a predicate verb appears in Haghighi et al., 2005), in addition to the generic syntactic features for the verb (Gildea and Jurafsky, 2000;Pradhan et al., 2004). As a result, few researchers investigated a feature engineering that could generalize across different syntactic configurations.
The lack of improvement in performance on the commonly used test data, such as Penn Treebank section 23, between 2005 and 2007 motivated SRL researchers to seek help from the grammatical relations between a word and its head within the framework of dependency grammar (Johansson and Nugues, 2007a,b;Surdeanu et al., 2008). However, the shift in feature representations, aiming to show that dependency grammar was more suitable for the feature design, was not The author would like to thank the reviewers for their valuable comments. The author would also like to thank Professors Steven Abney and Richmond Thomason and Mr. Terry Szymanski for their help.
Copyright 2009 by Li Yang convincingly supported by the results from CoNLL-2008. As a matter of fact, the researchers concluded, post CoNLL-2008, that determining the right syntactic/semantic knowledge for the SRL task still remained an open question (Surdeanu et al., 2008) and hence would be further explored on CoNLL-2009. The preceding conclusion and the trend in feature engineering indicate that finding the right syntactic/semantic knowledge remains a challenge. At the same time, how to generalize across different syntactic configurations involving the same verb remains an open question as well.
To provide a satisfying solution to the preceding question, we first identify the syntactic configurations that make it challenging for SRL researchers to create generalizable features. These configurations involve moved or displaced arguments. Then, relying on the knowledge of argument structure of the verb and on the context dependence among the core semantic arguments defined in the theory of argument realization (Levin and Hovav, 2005), we design the base argument configuration (BAC) features that generalize across the challenging syntactic configurations. Together with two-levels of backoff features, the BAC features effectively help with the argument role labeling subtask.

Basic definitions: core and non-core semantic arguments
We begin with defining the major terms in this paper. Following PropBank (Palmer et al., 2005), the present work defines a semantic argument as one of the constituents that participate in the event that the predicate is involved in, including the syntactic arguments, adjuncts, modal verbs, negation adverbs, and discourse markers. The core semantic arguments correspond to the syntactic arguments, and the non-core semantic arguments refer to the adjuncts, modal verbs, and discourse markers. In the rest of the paper, any mention of argument refers to semantic argument, unless otherwise specified. And we refer to the argument in syntactic structures specifically syntactic argument.

The challenge
We motivate the current project by depicting the fact that little had been done to design features to generalize across different syntactic structures that a verb occurs in. In the past eight years, SRL researchers have increased the number of features from the initial five in Gildea and Jurafsky (2000) to over twenty in later work such as (Pradhan et al., 2004;Toutanova et al., 2005;Haghighi et al., 2005;Surdeanu et al., 2007). While a large portion of these provide rich representation of the lexical and syntactic features of the arguments, more and more features are engineered towards specific syntactic structures. Punyakanok et al. (2005) imposed seven constraints on seven types of surface syntactic structures. Having found that the previous systems had not properly handled the cases where two consecutive arguments could take the same semantic role in relative clauses and the noun phrase it modifies, Toutanova et al. (2005) created the repeated core argument label feature. Moreover, Haghighi et al. (2005) came up with the projected path feature to handle the moved subject in the control-verb structures. Interestingly, it was Gildea and Jurafsky (2000) who first started this trend of tailoring features to specific syntactic structures. In their seminal work, Gildea and Jurafsky (2000) created the voice feature to distinguish between the passive and active configurations involving a verb. In the following, we illustrate the base argument configuration feature that generalizes across different surface syntactic structures. Intuitively, features generalize over different structures may increase the coverage of the classifier. In the next section, we introduce the basic syntactic and lexical semantic concepts that lend background to our solution to the challenge.

Syntactic and lexical semantic background
In this section, we introduce the syntactic and lexical semantic knowledge that allow us to create a feature design that generalizes over the diverse syntactic variations involving a verb.

Argument structure and base argument configuration
The arguments of a verb refer to its subject/external argument and complement(s)/internal argument(s). Argument structure refers to the number of external and internal arguments that a particular predicate verb requires in a clause (Carnie, 2002, p. 166). This definition indicates that the argument structure of a verb not only specifies the number of syntactic arguments it takes but also determines their positions in the structure. Based on the number of possible arguments and their positions in the argument structure of a verb in a single clause, Quirk et al. (1985, p. 53) summarize three main types of argument structures, including the 2-element structure Subject-V, the 3-element structure Subject-V-O/AdjP/PP complement, and the 4-element structure Subject-V-O-O/AdjP/PP complement. It is possible that a verb has more than one argument structure. In this paper, we name the subset of argument structures of the verbs in active voice learned from Treebank as base argument configuration.

Variations from base argument configuration
From Treebank, our system also learns four types of structures where some argument is either moved or displace so that they are not present at the position in the base argument configuration they are expected to be at. Argument movement and displacement cause structural variations from argument structures or base argument configurations. This is the reason why previous SRL systems created specific features to account for the variations, such as the passive feature to account for the arguments of a passive verb. The knowledge of these structures forms the basis of our system. The structures include the ones involving movements, displaced arguments, shared arguments, and extra arguments.

Lexical semantic background: context dependence
In addition to the preceding syntactic background, the current system also draws up the theory of argument realization (Levin and Hovav, 2005), the updated Linking theory (Levin and Hovav, 1996). The latter has been the linguistic theoretical basis since (Gildea and Jurafsky, 2000;Gildea and Palmer, 2002). Levin and Hovav (2005) extended the original linking theory in many aspects. We utilize the context dependence among the semantic roles of core semantic arguments in the updated theory by enforcing the context dependence constraint as follows.

Context dependence constraint:
The semantic roles of the core semantic arguments of a given verb that are dependent on each other are realized through a specific syntactic configuration of the core semantic arguments of the verb, within which the positions of the arguments assume fixed positions relevant to the verb and other arguments.

Feature design
With the foregoing background knowledge, we realize that both base argument configuration and context dependence concern the core semantic arguments and that the moved and displaced arguments are also core semantic arguments. Therefore, we decide to create different features for core and non-core arguments. For the core arguments, we propose the base argument configuration features and two levels of backoff features. For the non-core arguments, we extract a generic set of commonly used features which happened to be the second level of backoff features.

Base argument configuration features
The base argument configuration (BAC) feature is a sequential listing of the core semantic arguments of a predicate verb corresponding with their positions in the verb's base argument configuration/argument structure, with the verb inserted at the correct position in the listing. The BAC feature not only captures the base argument configuration of the verb but also imposes the context dependence constraint by ensuring all realized arguments to be present at their expected positions.

Level-II feature:
phrase type -the syntactic category of the constituent path -the path from the constituent to the predicate voice -the voice of the clause where the constituent is in position -position with respect to the predicate head word POS -POS of the head word of the constituent first word/POS -the first word/POS of the constituent last word/POS -the last word/POS of the constituent predicate -the predicate verb itself parent head word/POS -the head word of the constituent's parent node and its POS right sibling phrase type/head/POS -the syntactic category of the constituent's right sibling node, its head word, the head's POS left sibling phrase type/head/POS -the syntactic category of the constituent's left sibling node, its POS and head word  Create an instance for the candidate argument-verb pair.

5.
Extract the Level-II features for this instance. 6.
Classify the instance based on the features. 7.
Assign a core, non-core, or non-argument label to the candidate argument. 8.
If core argument is assigned, then add this argument to the core-argument list of the current verb. 9.
If non-core argument, then assign the argument to the non-core argument list of the current verb. 10. Done. 11. Done.

Procedures to determine candidate arguments:
Input: A constituent Current verb 1 If the parent of the current constituent is the verb, then return yes 2 Otherwise,check if the constituent is a moved argument, displaced argument, antecedent of relative clause, or in a co-ordinated structure 2.1 If yes, then use heuristics to determine if the constituent is an argument of the verb. If yes, then return yes. 3. Return no.

Level-I features: handling unrealized core arguments
If any of the argument of the verb is unrealized, the Level-I representing the current arguments and their positions in the base argument configuration is extracted.

Level-II features: handling unknown verbs
Both BAC and Level-I features are centered on the predicate verb. There are cases where the predicate during testing is unknown. In these cases, a set of generic features commonly used in literature are used (Pradhan et al., 2004;Toutanova et al., 2005). This set consists the features shown in Figure 1. The Level-II features are also extracted for non-core arguments, according to their expected positions in the verb's argument structure.

System architecture
The currents system proceeds with an argument identification phase and an argument classification phase.

Argument identification module
Unlike existing SRL systems that explicitly or implicitly identify the constituents as arguments or non-arguments, the current argument identifier performs a three-way identification. That is, a candidate is identified as either a core argument, non-core argument, or non argument. The purpose of identifying the arguments as core and non-core arguments is to be able to assign different sets of features to them. The argument identifier is summarized in Figure 2.

Main Procedures:
Input: ✔ A sentence ✔ lists of core arguments, one for each verb, ordered by their current positions in the sentence ✔ lists of non-core arguments, one for each verb 1. For each verb, do 2.
Normalize the order of the core arguments in its core-argument list.
3. For each verb in the current sentence, do: 1.
For each argument of the verb, do: 2.
Create an instance for the argument-verb pair. 6.
If the current argument is a core argument, then do 6.1. Extract the BAC feature for the current instance. 6.2 If there is unrealized argument(s), extract the Level-I feature 6.3. If the verb is new verb, extract the Level-II features. 6.4 Go to step 8.

7.
If the current argument is a non-core argument, then extract the Level-II features for the current instance.
8. Classify the current instance using the extracted feature(s) and assign the semantic role to the argument. 9.

Argument classification module
The argument identification module returns a list of core arguments and a list of non-core arguments for each verb. For each non-core argument, the classification module extracts the Level-II features, classifies them, and assigns them the corresponding semantic role label. For the core arguments, the critical step is to normalize the arguments' positions. If any argument appears in one of the structures involving moved or displaced argument, the argument will be returned to its originating position based on the built-in knowledge of the moved/placed argument positions in these structures.

Experiments, results, and discussion
We trained a three-way argument identifier using logistic regression with L2 regularization. We also trained KNN and L2 logistic regression argument classifiers with three different settings of features. WSJ section 24 was our development data. We performed three sets of experiments with WSJ section 23. We report the results from the three sets of experiments with WSJ section 23 in the following.

Experiments with argument classification only
This set of experiments assumed that core and non-core arguments were given and only performed the argument role assignment task. The goal of this set of experiments is to test our system and feature designs for the role assignment task. Table 1 lists the argument classification results from this set of experiments. The classifiers in Experiment #1 were trained/tested using the BAC features for each argument. Level-I features were added to the classifiers in Experiment #2. And Level-II features were added to Experiment #3. The Baseline classifier was trained/tested using only the Level-II features. The re-ranking system was an re-implementation of the jointinferencing model in Toutanova et al. (2005) by Surdeanu et al. (2007), one of the top models in the CoNLL-2005 SRL shared task. Surdeanu et al. (2007) also built system using combined strategies, which remains the state-of-the-system using constituency grammar representations. Since the KNN classifier did a slightly better job than the LR model, all scores about our system refer to those of the KNN models in the following. We could make the following conclusions from our experiments. The BAC features alone were effective because Experiment #1 gets higher precision, recall, F-measures than both the baseline and the re-ranking model. The Level-I features helped Experiment #2 get higher recall than the combined strategies, while increasing the overall F-measure. The Level-II features expectedly helped Experiment #3 with recall and F-measure, showing the increased coverage of the Level-II features. One more note we would make for this set of experiments is that the argument classifier per se is quite effective and consistent as will be shown in the next two sets of experiments.

Identification and classification with gold parses
In this set of experiments, we started with an LR argument identification module that identified the core and non-core arguments from the Treebank gold parses. We then ran the top performing KNN argument classifier from Table 1 on the automatically identified core and non-core argument lists. Table 2 lists the results. We could make the following observations from the results. Although the identification and classification results are higher than the baseline system, the performance of the argument classifier is greatly affected by that of the argument identifier. However, judging by the percentage classification correctness of 89.87, 90.52, and 90.20 in precision, recall, and F-measure from the identification results, the argument classifier performs consistently because the percentage correctness in all three measures matches that of the top performing argument classifier in Table 1.

Identification and classification with automatic parses
In this set of experiments, we fed the argument identifier with automatically parsed input using Charnaik parser McClosky et al. (2006). We then ran the top performing KNN argument classifier from Table 1 on the identified core and non-core arguments. The results are shown in Table 3. We observed similar trends as in the preceding set of experiments. The argument classifier is affected greatly by the identifier. The fact that the final system F-measure is several percent lower than the state-of-the-art system makes the argument identification results unacceptable. Despite the performance of the argument identifier, Table 3 shows that the argument classifier remains consistent because the percentage correctness of 89.26 89.93 and 89.60 in precision, recall, and F-measure from the identification results again matches that of the best argument classifier from Table 1.

Related work
As far as we know, the current system is the first one that specifically explores a feature design that generalizes across different syntactic configurations that a predicate occurs in. It has been a long history that SRL systems have tried to use the dependence among semantic arguments, such as (Gildea and Palmer, 2002;Xue and Palmer, 2004;Toutanova et al., 2005). However, these systems represent the dependency by simply listing all the arguments of a verb without leaving out the noncore arguments because context dependence only exists among the core arguments. The current system distinguish between the core and non-core arguments, which makes the BAC features high effectively, as shown in the experiments. At the same time, these systems do not identify moved or displaced arguments. Representing context dependence by simply listing the moved or displanced arguments without resolving their original positions may not precisely represent the dependency in the argument structure of the verb. Vickrey and Koller (2008) applied sentence simplification to transform each clause in the sentence to a simple sentence before assign semantic roles to the constituents. They had to write about 150 rules to transform the sentences. We rely on the knowledge of the structures involving moved and displaced argument to reconstruct the original positions of the arguments. But, our set of heuristics is far fewer than 150. In addition, Vickrey and Koller (2008) did not perform argument identification while we do.

Conclusion and future work
In this paper, we showed that the base argument configuration features, based on the knowledge of argument structure, on the constraint imposed by context dependence defined in the theory of argument realization, and on the knowledge of moved and displaced core arguments, and the two levels of features are robust in terms of solving the argument classification task. However, we have yet to improve the argument identifier in order to achieve the overall system performance.