{'fraction_non_alphanumeric': 0.027772207740124323, 'fraction_numerical': 0.009725285742931623, 'mean_word_length': 4.810952519662103, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Toward the automatic extraction of knowledge of usable goods

Knowledge of usable goods (e.g., toothbrush is used to clean the teeth and treadmill is used for exercise) is ubiquitous and in constant demand. This study proposes semantic labels to capture aspects of knowledge of usable goods and builds a benchmark corpus, Usable Goods Corpus, to explore this new semantic labeling task. Our human annotation experiment shows that human annotators can generally identify pieces of information of usable goods in text. Our first attempt toward the automatic identification of such knowledge shows that a model using conditional random fields approaches the human annotation (F score 73.2%). These results together suggest future directions to build a large-scale corpus and improve the automatic identification of knowledge of usable goods.


Introduction
A rich body of information extraction techniques focuses on acquiring knowledge from a huge amount of text data (Nickel et al. 2016). This allows large-scale knowledge bases to cover a broad range of knowledge. However, an important subfield of knowledge is not fully addressed: knowledge about use of objects such that hand sanitizer is used to kill bacteria and dental floss is used to remove plaque. Every object that humans create has its own purpose and function. We call these pieces of information knowledge of usable goods. Knowledge of usable goods is ubiquitous and in constant demand. People use search engines to find information on effect caused by using a new product, its proper way to use, and so on.
Knowledge sources that contain such information would also be beneficial for various kinds of natural language processing tasks, such as question answering systems and textual entailment. However, knowledge of usable goods is not thoroughly covered by current knowledge bases because these resources focus on entities (e.g. person or organization) and their relations (e.g. Is-PresidentOf). Section 4.3 shows the gap between kinds of knowledge available in the current knowledge bases and the ones that we aim to acquire.
To fill in this gap, this study proposes a set of semantic labels to capture knowledge of usable goods and builds a benchmark corpus, Usable Goods Corpus, to explore the automatic extraction of such knowledge. This work begins with focusing on information of health care and household goods such as air freshener, rice cooker, and nasal strip.
We assume that one of the most important aspects of knowledge of usable goods is about effects caused by using/consuming them as in (1). 1 (1) a. Fish-oils ... are known to reduce inflammation in the body, ... (fish oil) b. Alcohol-based hand sanitizers are more effective at killing microorganisms than 1 Throughout this paper, each typewriter word in a round bracket (e.g. toothbrush) indicates a name of a usable good that corresponds to the title of Wikipedia article. Humans can easily understand what the effects of these goods are: fish-oils reduce inflammation in the body (1a), hand sanitizers kill microorganisms (1b), BB cream tints and moisturizes skin (1c), and dental floss eliminate plaque (1d). However, the automatic extraction of such knowledge is challenging in that these effects can be expressed in various ways such as a verb phrase (1a), gerund (1b), noun phrase (1c), and clause (1d). This poses a problem that superficial linguistic patterns would not help identifying these kinds of expressions. To gauge difficulties of the automatic acquisition of these pieces of information, we conduct human annotation (Section 4) and automatic identification experiments (Section 5).
The major contributions of this work are: (i) We define a set of semantic labels to capture knowledge of usable goods, suggesting a new semantic labeling task. (ii) We experimentally build a benchmark corpus (Usable Goods Corpus) to explore the automatic extraction of knowledge of usable goods. The corpus and guidelines will be available when this paper is presented. (iii) We present our initial attempts toward the automatic extraction of such knowledge using a sequence labeling method. The results in this experiment provide measures to estimate the complexity of this task and suggest future directions to build a large-scale corpus.

Related work
To our knowledge, there is no resource that focuses on knowledge of usable goods. There are manually constructed and relatively accurate lexical resources such as WordNet (Miller 1995) and FrameNet (Baker et al. 1998), but their coverage is inevitably limited and these ontologies do not contain knowledge of our interest. Current large-scale knowledge bases focus on knowledge of entities and their relations, but the coverage of knowledge of usable goods is still sparse as shown in Section 4.3. OpenIE systems ) such as TextRunner (Etzioni et al. 2008) and ReVerb ) extract a large number of relations such as treadmill, burns, more calories using lexico-syntactic patterns from massive corpora drawn from the Web. Though these systems cover a wide variety of relational expressions, they do not intend to extract information of usable goods.
As for extracting information of objects, there is a body of research on the acquisition of telic and agentive roles in the context of generative lexicon theory (Pustejovsky 1991). Pustejovsky proposes qualia structures that define prototypical aspects of word's meaning (Pustejovsky et al. 1993). Of four semantic roles in the qualia structures, the telic role describes the purpose or function of an object (e.g. read is a typical telic role for book). Computational approaches are suggested to automatically extract expressions of this role from text (Yamada et al. 2007, Cimiano andWenderoth 2007), but these models tend to focus on taking paraphrases of "using X", rather than the expressions of purpose or function of objects. While the telic roles cover a broader range of expressions (probably due to the unspecified definition of telicity in the original theory), our work focuses on effects caused by using/consuming objects, standing as complementary to these previous studies.
Information extraction research in biomedical domains concerns effects caused by using drugs such that drug X causes adverse effect Y (Gurulingappa et al. 2012). This kind of information may overlap with what we aim to acquire, but ontologies in these studies are domain-specific such as protein interactions and adverse effects, contrary to our interest, which is more generic.
In summary, neither existing resources nor methods focus on knowledge of usable goods. In the next section, we propose a set of semantic labels that captures aspects of knowledge of usable goods.

Semantic labels for capturing knowledge of usable goods
To capture aspects of knowledge of usable goods, we define semantic labels as in Table 1 based on observation of 25 Wikipedia lead sections on health care and household goods. The Wikipedia lead 4 is normally a summary of its most important contents, and therefore it may allow us to get rich information from relatively small amount of data.
As shown in (1), we assume that one of the most important aspects of knowledge of usable goods is about effects caused by the use of goods. We also observe that there are various kinds of information that express degree/certainty of effects and conditions for the occurrence of effects. 4 It is also known as the introduction of a Wikipedia article, the section before the table of contents and the first heading. Table 1 are intended to capture these kinds of information. In addition to these semantic labels, we define a label Target for name and other expressions that refer to a usable good in the article. Names of usable goods essentially correspond to titles of Wikipedia articles, which refer to the topic of the text. Figure 1 shows how these labels are assigned to pieces of information about fish oil.

Semantic labels in
The annotation guidelines are designed to increase consistency. We define rules for segmentation in the guidelines, along with definition and examples of each label. To capture various linguistic expressions as illustrated in (1), we do not define a particular syntactic category for each label. All labels can take any type of linguistic constituent, but function words that do not contribute to the meaning are not included in each segment to avoid inconsistency. For example, we ask annotators mark define the eyes in Eyeliner is a cosmetic used to define the eyes as Effect (i.e., to is not included).
The set of semantic labels in Table 1 proposes a new semantic labeling task. To gauge the complexity of this task, we conduct human annotation experiment (Section 4) and automatic identification experiment (Section 5) as follows.

Annotation
We conduct a pilot annotation experiment to measure the complexity of this task. Measures of inter-annotator agreement and distributional analysis of the annotated data provide indications to improve the annotation schema for building a large-scale corpus in the future. This pilot corpus is also used for the automatic identification in Section 5. The following describes our annotation experiment in details.

Data: snippets from Wikipedia leads
We collect 200 English Wikipedia articles for annotation. Each article is about a health care or household goods such as toothpaste, tea cosy, and dishwasher. We choose these items using Ama-  zon categories and products lists. 5 All of chosen items are expressed as common nouns. We exclude any company-specific product. We extract the lead section of each Wikipedia article for annotation. We use at most the first 5 sentences of the lead to even out the number of sentences, ending up 792 sentences in total from 200 lead snippets.
Each annotator annotates same 100 snippets using brat (Stenetorp et al. 2012). Figure 1 shows an example of annotation. In addition to these 100 snippets, one of the two annotators annotates another 100 snippets, resulting in 200 annotated snippets. We use this set of 200 annotated snippets as the gold standard dataset in the following automatic identification experiment.

Evaluation
Two annotators were given the guidelines and a short training on texts not included in the corpus. Their task is to annotate linguistic expressions that correspond to the semantic labels in Table 1.  agreement. We compute these scores in two ways: (i) strict match: the starting and ending of the segment to be the same, (ii) lenient match: the starting and ending of the segment do not have to be the same but they overlap. We obtain Kappa coefficient of 0.57 in the lenient match, suggesting moderate agreement (Landis and Koch 1977). F score in the strict match (micro average 36.8%) seems to be reasonable because we give annotators unparsed raw text to explore the range of linguistic expressions. Most segmentation disagreements occur in deciding whether to include function words (e.g. to protect skin or protect skin).
In addition, there are label disagreements accounting for 20% of segment pairs that either partially or completely match. For example, one annotator marks hair and skin care in (2) as Effect and the other does so as Means of Use, where both labels seem to be appropriate.   (3) shows an example of disagreement between Version and Composed of.
(3) A wet wipe ... is a small moistened piece of paper or cloth ... (Wet wipe) Paper and cloth in (3) could be Version of wet wipe, but they are also materials that compose of wet wipe. Both Version and ComposedOf are valid in this example. These examples of label disagreement suggest that single-label annotation would not be able to sufficiently capture the knowledge of usable goods. Allowing multi-labeling would be one direction for further improvement.

The distribution of the annotated data
We conduct distributional analysis to examine the extent to which the proposed semantic labels capture information of usable goods. Table 3 breaks up numbers of the annotated instances by two annotators. Effect results in the most frequent one, suggesting its significance at least in the domain of health care and household goods.
On the other hand, there are a few number of instances for Certainty of Effect, Degree of Effect, Null Effect, Part of, Location, Time, and User. This may due to the content of the Wikipedia leads. These kinds of more precise information would usually appear after the lead section. 6 We further examine the syntactic distribution of Effect instances as in Table 4. The majority of Effect instances are represented as verb phrases and there is a variation in those instances such as darken the eyelids (kohl), minimize shininess caused by oily skin (face powder), tones the face (face powder), reflect light at different angles (glitter) and so on, in addition to typical causal expressions such as causes anesthesia (anesthetic), prevent snoring (nasal strip), and promote oral hygiene (toothpaste). An example of noun phrase in Table 4 suggests an interesting problem in that lacquer itself is a usable good but also means effect caused by using a nail polish. This kind of information structure has not been addressed in previous work on information extraction.
Overall, we find that 81.8% of instances occur with Target in the same sentence. The remaining cases involve long-distance dependencies across the sentence. This distribution suggests that we do not need to annotate the relation between Target and each label and we could exploit these inter-sentential relations in the automatic identification task. The following Section 5 shows our automatic identification experiment using this distributional property.

Comparison with current knowledge base
The above human annotation experiment shows that Wikipedia leads contain a reasonable amount of information on effects caused by us-6 Besides these semantic labels, there are other descriptions on the manufacturing process and history of usable goods as in (4).
(4) a. (herbal distillate) ... obtained by steam distillation or hydrodistillation (herbal distillate) b. Modern perfumery began in late 19th century with the commercial synthesis (perfume)  For comparison, we use 100 usable goods in our corpus such as ice pack, hand sanitizer and perfume. We then manually select 4 out of 39 pre-defined relations in ConceptNet that could be associated with effect expressions such as Used For, Capable Of, Causes Desire, and Causes. Of 100 usable goods, 27 usable goods have pieces of knowledge that are expressed with the above relations such as hand sanitizer, Causes, clean hand and Toothpaste, Capable Of, help remove plaque .
In short, though ConceptNet contains information of our interest, the coverage is still not sufficient (27/100 usable goods). The automatic extraction of information of usable goods would help populate this kind of knowledge base. The next section shows our initial attempt toward the automatic extraction of knowledge of usable goods.

Sequence labeling model for identifying information of usable goods
This section presents our experiment for automatically identifying information of usable goods. The results provide baseline measures for this new semantic labeling task and suggest potential directions for improvement. Section 4.3 shows that almost all instances in our corpus occur with Target in the same sentence. We exploit this distributional property by using Target words as a cue to find information of usable goods and pose this task as a sequence labeling problem. We use Conditional Random Fields (CRFs), a popular approach to solve sequence labeling problems (Lafferty et al. 2001). CRFsuite 7 is used as an implementation of CRF for our purpose.

Experimental Settings
The training and test data consists of 792 sentences from 200 Wikipedia snippets (see Section 4.1). We select the four most frequent labels in the corpus, Effect, Means of use, Composed of and Version, for evaluation.
For the data pre-processing, we first parse the raw text and assign a part of speech tag and a named entity tag to each word using Stanford CoreNLP (Manning et al. 2014). Then we add a semantic label to each word with BIO format (Beginning, Inside and Outside).

Features
Features shown in Table 5 are used for training. We use these features within a window of ±3 around the current word. Some of these features are used in combination with another feature as shown in Table 5.
In addition to standard features, we add three features to exploit the characteristics of this corpus: Target, Disease and Repeat. Target feature is true when the current word is same as the title of Wikipedia article. Disease feature is true when the current word is in a list of disease names that we create using Freebase (Bollacker et al. 2008). This feature is intended to capture effect expressions that include disease names such as provoke allergy and asthma symptoms (air freshener). Repeat feature is true when the current word has already been appeared in the sentence. This feature is intended to capture a parallel structure that is often used  to express Version and Composed of.

Evaluation
We compute precision, recall and F1 measure using ten fold cross validation. We compute these scores in two ways, lenient match and strict match as in the human annotation experiment (see Section 4.2). Table 6 shows results. F score in the lenient match (73.2%) approaches the human annotation performance (81.9%). This suggests that the model is able to identify labels to some extent. For example, the model recognizes typical lexico-syntactic patterns such as be used to in (wallpaper) is used to cover and decorate the interior walls and be designed to in (rice cooker) is designed to boil or steam rice. Furthermore, the model captures various effect expressions such as an adjective phrase (5a), verb phrase (5b), and gerund (5c).
(5) a. Chandeliers are often ornate, and normally use... (chandelier) b. A diuretic is any substance that promotes the production of urine.
c. An espresso machine brews coffee by forcing pressurized water near boiling point... (espresso machine) On the other hand, the segmentation problem as discussed in the human annotation experiment influences the F score in the strict match (13.7%).
In sum, though there is the segmentation problem derived from the annotation, the results in the lenient match suggest that the model can identify information of usable goods to some extent. Improving the annotation schema and increasing the size of the corpus would be promising directions for future work.

Conclusion
This paper proposes semantic labels to capture aspects of knowledge of usable goods. We design annotation schema and build the benchmark corpus, Usable Goods Corpus, based on the proposed semantic labels. Our human annotation experiment shows that (i) while there is the segmentation mismatch problem, human annotators can generally identify pieces of information of usable goods, and (ii) Wikipedia leads contain a reasonable amount of information on effects caused by using goods in contrast to the coverage of the current knowledge base. The automatic identification experiment shows that despite of the influence of the segmentation problem in the human annotation, the model can to some extent identify pieces of information of usable goods.
Our next steps are to alleviate the segmentation problem and increase the corpus size. With these goals in mind, we plan to revise the annotation schema as follows: (a) Some semantic labels do not seem to be important as seen in the statistics in Table 3. Reducing the variation of the semantic labels is a reasonable direction. (b) Defining a syntactic category for each label and giving annotators/models parsed text would increase consistency in the segmentation. (c) These simplifications (a,b) would allow us to  Table 6: 10-fold cross-validation try crowdsourcing annotation to increase the size of the corpus.