{'fraction_non_alphanumeric': 0.030492875755391696, 'fraction_numerical': 0.011809059156178966, 'mean_word_length': 5.194368131868132, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The TextCap Semantic Interpreter

The lack of large amounts of readily available, explicitly represented knowledge has long been recognized as a barrier to applications requiring semantic knowledge such as machine translation and question answering. This problem is analogous to that facing machine translation decades ago, where one proposed solution was to use human translators to post-edit automatically produced, low quality translations rather than expect a computer to independently create high-quality translations. This paper describes an attempt at implementing a semantic parser that takes unrestricted English text, uses publically available computational linguistics tools and lexical resources and as output produces semantic triples which can be used in a variety of tasks such as generating knowledge bases, providing raw material for question answering systems, or creating RDF structures. We describe the TEXTCAP system, detail the semantic triple representation it produces, illustrate step by step how TEXTCAP processes a short text, and use its results on unseen texts to discuss the amount of post-editing that might be realistically required.


Introduction
A number of applications depend on explicitly represented knowledge to perform basic tasks or add customization to existing tasks. Improving the quantity and quality of the knowledge contained in knowledge bases could lead to the improved performance of many applications that depend on knowledge and inference such as: • Generating scientific or educational explanations of natural or mechanical systems and phenomena (Lester and Porter, 1997), • Question answering systems (Clark et al., 2001) that use reasoning to solve problems rather than looking up answers, • Multimodal information presentation systems that depend on specific real world knowledge in order to describe or refer to it for audiences (Callaway et al., 2005;Stock et al., 2007).
These systems have typically relied on hand-built and domain specific knowledge bases requiring years of effort to produce. The need to speed up this process as well as make the resulting representations more consistent are well-known problems that have yielded a number of potential solutions (Blythe et al., 2001;Reiter et al., 2003;Carenini et al., 2005;Barker et al., 2007), but large scale, domain independent, and fully automatic knowledge acquisition on unrestricted text is still in its infancy.
Over the last decade research in applied computational linguistics has extended the various components necessary for semantic parsing, but have tended to focus on increasing the measurable performance of individual subtask in isolation (e.g., parsing, anaphora resolution, semantic role labelling, and word sense disambiguation) rather than on an entire end-to-end system. Meanwhile, theoretical CL research has examined issues such as underspecification, scoping and reference resolution in discourse contexts, but has set aside issues such as large-scale robustness, ontology integration and evaluation which are vital for applied uses of semantic parsing.
In this paper we discuss an implementation to automatically extract explicitly coded conceptual and ontological knowledge from unrestricted text using a pipeline of NLP components, as part of the STEP shared task (Bos, 2008). The TEXTCAP system performs the basic steps towards this task by gluing together an off-the-shelf parser with semantic interpretation methods. It is intended to be a test case for (1) establishing baseline performance measures for semantic parsing and (2) determining what degree of post-editing might be necessary in real-world environments.
Because major components of such a system would not be tailored towards the semantic parsing task, we would rightly expect its output to be imperfect. This problem is analogous to that facing machine translation decades ago, where one proposed solution was to use human translators to post-edit automatically produced, low quality translations rather than expect a computer to independently create high-quality translations. One aspect of this research is thus to investigate how much post-editing would be required to convert the system's output to usable semantic triples.
Finally, this paper presents the results of TEXTCAP on the 2008 STEP shared task corpus, giving specific comments about the difficulties in encountered. Although not a formal evaluation, we were satisfied with its performance in terms of accuracy and efficiency for helping humans post-edit semantic triples.
2 System Description TEXTCAP performs basic steps towards the task of converting free text into semantic triples by gluing together an off-the-shelf parser with ad-hoc semantic interpretation methods. TEXTCAP parses a document into Penn TreeBank form and then traverses each syntactic parse tree performing a series of step-by-step tasks such as discourse parsing, clause separation, word sense disambiguation, anaphora resolution and semantic role labelling. Ad hoc rules then create a set of triples from the resulting semantically-enhanced parse tree.
TEXTCAP first uses the domain-independent Charniak parser (Charniak, 2000) to convert sentences in the source document into a sequence of syntactic parses. It then applies syntax-based discourse parsing rules (such as Soricut and Marcu (2003)) to reduce coordinate, subordinate, and relative clauses into coindexed, simpler sentence parses headed by single verbal relations.
It then marks for grammatical roles (subject, object, etc.) and syntactic features (e.g., passivity) before using a simple anaphora resolution algorithm based on those features and a word sense disambiguation algorithm grounded in WordNet (Fellbaum, 1998) senses that helps determine additional features such as animacy. A two-pass method is applied where first monosemous words are assigned senses, and then remaining senses are selected together with verb types (TEXTCAP uses ad hoc rules rather than current verb taxonomies like FrameNet). Selectional restrictions from the verb type then allows for labelling of peripheral grammatical roles as semantic roles. Finally, entities representing specific objects are marked with ontological relations and discourse relations are realized between individual verbal relations.
The end product of TEXTCAP is thus a list of coindexed semantic triples representing the explicitly recoverable semantic content of the input text.

Text Processing Components
Corpus methods underlie many of the recent improvements in a wide array of generic NLP tools. For instance, the introduction of large-scale lexical and syntactic resources like the Penn TreeBank (Marcus et al., 1993) have led to highly accurate, domain independent parsers (Collins, 1999;Charniak, 2000). Wide-coverage anaphora resolution systems process references across multiple sentences, and recent work on anaphora resolution by Poesio and Kabadjov (2004) describes itself as the first such system which can be used off-the-shelf.
Word sense disambiguation (Gliozzo et al., 2005), often based on term frequency analyses of large annotated corpora, can help localize search in a particular area of a knowledge base to find the most related concepts and instances. Semantic role labelers (Gildea and Jurafsky, 2002;Yeh et al., 2006) annotate what role each entity has in relation to its local man verb, and can provide additional clues for disambiguating words and locating them in an ontological space.
In addition to lexical and semantic tasks, multi-sentence linguistic analysis such as discourse segmentation and parsing is needed to semantically label the roles of verb phrases in relation to one other. Soricut and Marcu (2003) presented a statistical system that automatically produces an analysis of the rhetorical structure that holds between sets of sentences or clauses at the paragraph level.
As a generalization, NLP research has been conducted separately and few attempts have been made to connect each of them into the longer chains and pipelines needed for more complete and deeper text processing such as is needed for tasks like knowledge acquisition. Additionally, most of these tools are intended to iteratively examine each sentence individually within a larger document. But often important linguistic phenomena cross sentence boundaries, yet are just as necessary to properly understand the semantic content of a document.

Knowledge Representation in TEXTCAP
A common method of representing semantic knowledge in the Knowledge Base community (Brachman and Schmolze, 1985;Clark and Porter, 1998) is through threeplace predicates, or triples, of the form (CONCEPT RELATION CONCEPT). A concept can signify either a generic entity or class, like "houses", or a particular instance, such as "my house at 35 Lincoln Avenue"; instances are coindexed to indicate they are the same entity in multiple predicates. Relations are typed according to domain, range and cardinality, and can also be marked as instances to indicate that they refer to specific events or properties that hold at a particular time place, etc.
Databases and knowledge bases can both be represented as large collections of triples. Knowledge bases differ from databases in that they are generally organized around a hierarchical taxonomy, or ontology, of both entities and relations, allowing for subsumption as inference and for knowledge to be separated into subgroups. Knowledge bases differ from ontologies in that, like databases, they also contain a larger set of specific knowledge (instances) that describes non-taxonomic relationships between members and instances of the ontology's concepts.
For instance, the sentence "My dog chases rabbits." talks about a specific instance of the class dog and its relationship to the generic class representing rabbits, perhaps represented as the triple (DOG492 CHASING RABBIT-ANIMAL). To know that this dog really is a member of what we consider as the class of all dogs, we would need to add an ontological triple such as (DOG492 INSTANCE-OF DOG). To represent the possessive grammatical relation in "my dog" we would need to agree on some particular person (an instance) to represent the speaker of the utterance (PERSON142 INSTANCE-OF PERSON) and then also add a relation to indicate possession (DOG492 OWNED-BY PERSON142). Because language is ambiguous compared to semantic triples, we wouldn't want the word "my" to always be mapped to the same relation, for instance, obtaining (PERSON366 OWNED-BY PERSON142) from the phrase "my friend".
Like concepts, relations can also have instances since they can refer to particular events with particular modifiers. For instance, in the sentence "My dog quickly chased rabbits yesterday." we would need to change the relation CHASING from the triple above to (DOG492 CHASING141 RABBIT-ANIMAL) to indicate its modifiers, perhaps with (CHASING141 SPEED QUICKLY) and (CHASING141 EVENT-TIME YESTERDAY). We would also need to indicate the taxonomic relationship between the two relations, (CHASING141 INSTANCE-OF CHASING).
Because over the years different research groups have created differing ontologies, it is important to have a common ontology (and arguably, mapping of lexical items to classes in that ontology) for purposes such as evaluative comparison, even if implementations that acquire semantic triples can use any available ontology.
In keeping with the practice of much recent large-scale NLP, TEXTCAP uses Word-Net (Fellbaum, 1998) as an underlying ontology and sense repository for generic classes, giving it the ability to leverage recent NLP tools that rely on it, such as for word sense disambiguation (Gliozzo et al., 2005). Thus given the sentence in Figure 1(a), we are interested in producing the semantic triples in (b) where generic entities and relations are grounded in WordNet.

Processing The Text
To illustrate how TEXTCAP works, we follow how it processes the following paragraph of newspaper text from the New York Times: Amid the tightly packed row houses of North Philadelphia, a pioneering urban farm is providing fresh local food for a community that often lacks it, and making money in the process. Greensgrow, a one-acre plot of raised beds and greenhouses on the site of a former steel-galvanizing factory, is turning a profit by selling its own vegetables and herbs as well as a range of produce from local growers, and by running a nursery selling plants and seedlings. The farm earned about $10,000 on revenue of $450,000 in 2007, and hopes to make a profit of 5 percent on $650,000 in revenue in this, its 10th year, so it can open another operation elsewhere in Philadelphia.

Figure 1: WordNet senses as generic entities and relations
We first normalize this from the form used by the Charniak and Collins parsers (which do no semantic role labelling and introduce some simplifications) into a corrected version following the original Penn TreeBank format. In the above parse, the following lines are normalized to mark grammatical subject and correctly mark the auxiliary verb: . . . (NP-SBJ (DT "a") (JJ "pioneering") (JJ "urban") (NN "farm")) (VP (VBZ "is") (VP (VP (VBG "providing") . . .

Performance on the Shared Task
Overall, TEXTCAP performed well for its intended purpose, but many limitations were encountered on unseen texts, as expected. Principally, word sense disambiguation and pronoun resolution initially caused significant problems in terms of robustness and the capabilities of these text processing steps were significantly downgraded in order that TEXTCAP could run to completion on all seven sets of unseen texts. Thus WSD was run only for WordNet noun senses and pronoun resolution was not run across sentence boundaries within each set. Additionally, the discourse parser lacked rules to correctly convert sentences #1 and #4 in set #5, so the input sentences were manually split in that case.
However, TEXTCAP was able to do a good job at producing semantic triples for every text, and the number of triples was proportional to the length of each sentence, as expected. The use of existing lexical tools and resources allows for more time to be spent on adding and correcting semantic mappings. Some necessary lexical tools are either not available or still limited in terms of accuracy, and some resources do not exist, for instance, there is no good ontological inventory of prepositions and how they should be mapped semantically. In general, overall accuracy (as measured by human inspection) was much better on shorter sentences.
The system performed poorly in some areas such as interpreting questions and quotations involving multiple sentences. Additionally, the structure of many of the triples that TEXTCAP produced were highly reflective on the original syntactic parses -it is not clear, for instance, that they would enable a question answering system to locate correct answers reliably. However, overall, we believe that post-editing of triples with TEXTCAP would provide a significant time speedup compared to manual knowledge engineering alone, and we are looking at methods of showing this empirically.
The following data represent the performance of TEXTCAP on the 2008 STEP shared task. Sentences were processed in an average time of 4 seconds each. Set #1 [1] "An object is thrown with a horizontal speed of 20 m/s from a cliff that is 125 m high." Notes: (a) the parser interpreted "m/s" as a plural noun; (b) source is a very vague relation; (c) cliff was correctly recognized as a relative clause subject. [2] "The object falls for the height of the cliff." Notes: (a) for was incorrectly intepreted as purpose ("object" would be animate); (b) of yielded the wrong relation; (c) it's unclear what should be the 3rd element of the triple for falling.
[2] "Once broad-scale electricity distribution spread to farms and country towns, use of wind energy in the United States started to subside, but it picked up again after the U.S. oil shortage in the early 1970s." Notes: Notes: This sentence was processed satisfactorily, but only when manually split due to missing discourse parsing rules. [3] "Over the past 30 years, research and development has fluctuated with federal government interest and tax incentives." Notes: This sentence was processed satisfactorily.
[4] "In the mid-'80s, wind turbines had a typical maximum power rating of 150 kW." Notes: This sentence had problems understanding the phrase "mid-'80s", perhaps as a result of the off-the-shelf parser being very generic.
[5] "In 2006, commercial, utility-scale turbines are commonly rated at over 1 MW and are available in up to 4 MW capacity." Notes: (a) the fact that someone rates turbines isn't the same as turbines carrying a rating; commonly wasn't interpreted correctly; (c) the last phrase after available wasn't mapped to anything.

Conclusions
We introduced TEXTCAP, a semantic parser which uses a combination of off-theshelf NLP technology and ad-hoc rules to produce semantic triples corresponding to the explicit semantic content in unrestricted text. We ran TEXTCAP on 7 sets of short text in the STEP 2008 Shared Task, and the system successfully generated triples for almost all inputs and provided, as we expected, a set of triples that while not fully correct, could be post-edited for accuracy and which should provide a significant speed up over completely manual production of semantic triples from text. On average, TEXTCAP processed a sentence from the corpus in about 4 seconds.
While TEXTCAP only captures explicit knowledge (but not commonsense knowledge, unmentioned knowledge, implicit relationships, etc.) it can save knowledge engineers time by providing reasonably accurate semantic representations of domain text. In future work we plan on improving methods of knowledge integration (e.g., ontology population), testing within real-world applications such as question answering systems, and empirically evaluating the time and accuracy for producing semantic triples via various methods.