{'fraction_non_alphanumeric': 0.031336817039290914, 'fraction_numerical': 0.007441460510649557, 'mean_word_length': 4.802110817941952, 'pattern_counts': {'":': 1, '<': 1, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 2, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 15, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Concept-Based Embeddings for Natural Language Processing

In this work, we focus on effectively leveraging and integrating information from concept-level as well as word-level via projecting concepts and words into a lower dimensional space while retaining most critical semantics. In a broad context of opinion understanding system, we investigate the use of the fused embedding for several core NLP tasks: named entity detection and classification, automatic speech recognition reranking, and targeted sentiment analysis.


Abstract
Concepts are critical semantics capturing the high-level knowledge of human language. As a way to go beyond the word-level analysis, representing and leveraging the concept-level information is an important add-on to existing natural language processing (NLP) systems. More specifically, the concepts are critical for understanding opinions of people. For example, people express their opinion towards particular entities such as products or sentiment aspects in online reviews, where these entities are mentions of concepts rather than just words.
As compared with words, the mentions of abstract concepts may be compounded phrases (either consecutive or non-consecutive) that are likely to form a large vocabulary. Furthermore, there might be semantic properties (e.g., relations or attributes) attached to the concepts, which increases the dimensionality of concepts. In short, using concepts is faced with the curse of dimensionality. On the other hand, information from only a single level does not suffice for a thorough understanding of human language, and meaningful representation is required at any point to encode the correlation and dependency between abstract concepts and words. In this thesis, we thus focus on effectively leveraging and integrating information from concept-level as well as word-level via projecting concepts and words into a lower dimensional space while retaining most critical semantics. In a broad context of opinion understanding system, we investigate the use of the fused embedding for several core NLP tasks: named entity detection and classification, automatic speech recognition reranking, and targeted sentiment analysis.
We first propose a novel method to inject the entity-based information into a word embedding space. The word embeddings are learned from a set of named entity features instead of merely contextual words. We demonstrate that the new word embedding is a better feature representation for detecting and classifying named entities from the stream of telephone conversations.
Apart from learning input feature embeddings, we then explore encoding the entity types (i.e., concept categories) in a label embedding space. Our label embeddings mainly leverage two types of information: label hierarchy and label prototype. Since our label embedding is computed prior to the training process, it has exactly the same computation complexity at run-time. We evaluate the resulting label embeddings on multiple large-scale datasets built for the task of fine-grained named entity typing. As compared with the state-of-the-art methods, our label embedding method can achieve superior performance.
Next, we demonstrate that a binary embedding of the named entities can help reranking the speech-to-text hypothesis. Named entities are encoded using a Restricted Boltzmann Machine (RBM) and used as a prior knowledge in the discriminative reranking model. We also extend the training of RBM to work with speech recognition hypothesis.
Finally, we investigate the problem of using embeddings of commonsense concepts for the task of targeted sentiment analysis. The task is also entity-centered. Namely, given a targeted entity in a sentence, the task is to resolve the correct aspects categories and corresponding sentiment polarity of the target. We propose a new computation structure of Long Short-Term Memory (LSTM) that can more effectively incorporate the embeddings of commonsense knowledge.
In summary, this thesis proposes novel solutions of representing and leveraging concept-level and word-level information in a series of NLP tasks that are key to understanding the opinion of people.

Introduction
The understanding of human language involves processing information beyond the words. A complete process should be capable of handling information from various levels such as concepts or thoughts. Concepts, as one of the most basic units of language, refer to aspects of the world that are used by people in their language. Concepts are constituents including mostly phrases such as noun phrases, verb phrases, adjective phrases and so on and so forth. In addition, the definition of concepts is broad enough to include mentions of named entities which can be deemed as concepts belonging to certain categories, e.g., location, organization, or person.
Recently, a variety of knowledge resources such as ConceptNet have been built from the huge unstructured information thanks to the rapid growth of online communities and social network. These knowledge resources t are based on concepts rather than single words and are in the form of either a directed or undirected graph with concepts being the nodes and their relations being the edges. Additional attributes or properties might also be incorporated in order to better represent the semantics of concepts. These concept-based resources might benefit many NLP systems via providing accesses to external knowledge.
On the other hand, concepts are playing a core role in understanding the opinions of people. For example, people tend to express their sentiment towards a particular targeted entity (e.g., purchased products) in online reviews. By our definition of concepts, these entities are actually mentions of concepts. Therefore, the opinion understanding system would benefit from the use of concept-level information. However, 1 due to the complexity of natural language, a complete analysis of people's opinions requires performing a series of subtasks such as concept extraction (or named entity recognition) and sentiment analysis, and it is hard to train a single end-to-end trained model that can perform well on all the subtasks of an opinion understanding system.
Instead, it typically needs to build a pipeline composed of a series of natural language processing modules. In this thesis, we choose to extend the use of concepts to three important modules in the whole context: named entity recognition, targeted sentiment analysis and ASR hypotheses reranking. These tasks are core components of a multimodal opinion understanding system which is capable of processing not only text data but also audios. One of the natural obstacle hindering concepts from being employed in large scale is that concepts are generally of high dimensionality. Hence, our solutions focus on generating and leveraging concept-based embeddings for solving concept-centered NLP problems.

Preliminaries
We first introduce the definition of concepts as well as two knowledge resources that have been widely adopted for NLP tasks. We then give a brief introduction of conceptcentered NLP tasks that we are interested in addressing in this thesis: named entity recognition, fine-grained named entity typing, targeted sentiment analysis, and ASR hypotheses reranking.

Concepts
• Concepts are used by people to describe the objects or actions perceived. Concepts might have different forms in different contexts. Admittedly, the surface form of concepts are composed of single or multiple words. For example, the concept small room is composed of two words small and room. Inflections of concepts are prevalent in natural language. Namely, the same objects or action might be referred to as different concepts due to many reasons including cultural differences, the existence of synonyms and acronyms, and so on. We could deem a concept as an abstract term having a canonical form as well as several surface 2 forms or mentions. For example, the concept of which the surface is "New York" might have mentions like "New York city" or "big apple".
• Named entities [11,13] can also be deemed as concepts. In the very first challenge of named entity recognition [11], named entities are defined as noun phrases in its surface form and refer to objects belonging to categories such as person, location, and organization. Although different challenges such as CoNLL 2003 shared tasks have extended the definition to involve other concepts, a standard definition of named entities has been constrained to consider only flat categories (i.e., non-hierarchical). Notably, a recent study on finegrained named entities has further extended the concern with named entities to involve a hierarchical structure where IsA relation between concepts is taken into consideration.

Concept-based Knowledge Resources
Recently, knowledge bases such as ConceptNet [14] have linked concepts to more structural semantic information, e.g., relations and attributes. These knowledge bases are built automatically by employing rule-based and machine learning techniques to extract facts from Web texts. Therefore, the coverage of concepts could keep growing over the time.

ConceptNet
ConceptNet is automatically built as a combination and filtering of many existing and ever-growing multilingual knowledge resources such as Open Mind. ConceptNet keeps updating itself according to the source knowledge bases. One notable feature of ConceptNet is that it contains a large number of relational knowledge. Namely, we could be able to know the relations between concepts. For example, from ConceptNet, it can be known that the white house (argument 1) is located (relation) in america (argument 2) and a type of (relation) building (argument 3).

Concept-centered NLP for Opinion Understanding
The evolution of NLP techniques clears shows a trend moving from word-level to concept-level (as illustrated in Fig. 1.1). Merely relying on word-level analysis of natural language is limited in the power of expressiveness since the natural language is composed of different levels of abstraction. It is, therefore, advantageous to use concepts for NLP tasks for that it allows for access to the rich semantic features of particular concepts. For example, collecting information about battery life and price is helpful when comparing the two products "iPhone X" and "Samsung S8".
The concept-level analysis has been adopted for NLP tasks such as sentiment analysis and named entity recognition. However, concepts are very sparse due to the 4 Figure 1.2: A pipeline system [2] illustrating the key components of an opinion understanding system. Tasks selected as the focus of this thesis are highlighted in red boxes.
overwhelming number of unique concepts as well as concept inflections and features.
A more compressed and effective representation is required to encode the concepts.
On the other hand, a dense and lower dimensional representation also facilitate the exploration of deep learning models that have recently gained great successes in many NLP tasks [17]. With the access to knowledge bases, we are able to augment NLP solutions with concept-level semantics.
On the other hand, understanding the opinions expressed by human beings is one of the ultimate objectives of NLP. As shown in Fig. 1.2, an opinion understanding system requires the composition of multiple NLP modules involving natural language concepts, from which we identify the following tasks that either requires performing concept-level analysis or can benefit from using concept-level information: • Named entity recognition (NER). Named entities are typically belonging to several semantic classes defined according to the particular interest of downstream applications. For example, a flight booking assistant system might require recognizing the location names corresponding to departure and arrival cities is the most critical step for retrieving the best matched flight. A typical setting of NER is composed of both tasks of detection and classification.

5
In general, an NER model might take as input a sequence of tokens as well as a predefined set of entity categories, and then outputs the boundaries and semantic types of named entities presenting in the input.
• Fine-grained named entity typing (FNET). Typical setting of NER models aims at resolving only the coarse-grained types of named entities. As an extension of the classification task in NER, FNET aims at recognizing the finegrained types of a named entity, which are typically organized in a tree-shape hierarchy [18]. A natural formulation of FNET is a multi-label multi-class classification problem of which one entity is assigned with multiple categories.
• Targeted sentiment analysis (TSA). The targeted sentiment analysis is another concept-centered NLP task resembling the task of FNET. The input to TSA is identical to FNET in which the region of the targeted entity is presumed known, and the ultimate goal is mapping the targeted entity to its sentiment aspects and polarity labels (e.g., positive and negative). More specifically, the task of TSA is composed of two subtasks: 1) aspect categorization and 2) polarity classification. For example, the desired outputs for a product review Pixel 2 Xl is cheaper than Iphone X are the sentiment aspect, price, and the corresponding sentiment polarity label, negative, given the target is Iphone X.
Apart from these concept-centered NLP tasks, we also extend the use of conceptbased information to automatic speech recognition (ASR) candidate re-ranking, which could augment the opinion understanding system with the capability to perform on data of multi-modality. We assume that concepts such as named entities deserve more attention than functional words for their critical roles for downstream tasks such as spoken language understanding systems or a multimodal sentiment analysis system. Therefore, given a set of ASR decoding candidates, we hope the entity-based information could help us select the candidate the makes fewer mistakes in recognizing important keywords such as person names and location names. 6

Challenges and Problems Faced by Conceptcentered NLP
In the past decade, great progresses have been made on concept-centered tasks, especially for tasks such as named entity recognition that have been studied for decades.
In general, whenever concepts are concerned in these NLP tasks, it is naturally desirable to have an effective representation as well as the corresponding computational structure to leverage the concept-level information.
For knowledge base such as ConceptNet that extracts conceptual facts from the Web automatically, the number of concepts will grow over time. On the other hand, the number of semantic properties (or relations) associated with each concept might also be an obstacle for employing conceptual analysis.
Therefore, a better representation of concepts basically requires tackling two problems: 1) producing a condensed representation of the original feature space, and 2) retain the semantics encoded in the raw data. Ideally, the compressed representation minimizes the dimensionality and information loss at the same time. However, it is hard to fulfill in practice, and a trade-off must be taken in such case.
Apart from an efficient and effective dense representation of concepts, leveraging conceptual information also needs well-designed computation structure in order to be integrated with information flowing from other levels (e.g., word level).
Most importantly, we believe using only information from a single-level does not suffice to achieve a complete analysis of the natural language. Therefore, the fusion of multi-level information is required for advancing the NLP research.

Contribution
The focus of this thesis is to leverage concept embeddings for concept-centered natural language processing tasks discussed in the last section and investigating on how to fuse the information of multiple levels.

7
• We investigate several novel solutions on fusing the concept-level and word-level information. We demonstrate the effectiveness of such fusion with multiple concept-centered NLP tasks.
• We propose a novel method to inject concept-based information into the word embedding space which then being used as effective features to recognize concepts/entities from input sentences.

Organization
The rest of this thesis is organized as follows: Chapter 2 surveys the background and existing work on concept embeddings and concept-centered NLP tasks; Chapter 3 describes our method on injecting entity-based information into word embedding space to help recognize named entities from telephone conversations; Chapter 4 describes the concept-category embedding used for resolve fine-grained categories of named entities; Chapter 5 presents our binary entity embedding for reranking ASR hypotheses; finally, Chapter 6 presents our work on leveraging commonsense concepts for analyzing sentiment with respect to targeted entities.

Background
In this chapter, we review the related work on NER, ASR reranking, TSA, as well as recent progress on learning concept or word embeddings.

Named Entity Recognition
Recognizing named entities from texts such as news articles is a concept-level NLP task has been studied by the community. Typically, recognition of named entity requires jointly resolving the location and category of a named entity in its context. In other words, the task could also be split into two tasks: detection and classification (or called Entity Typing). We start with introducing the related work on NER ranging from rule-based approaches of early days to most recent deep neural networks. We then extend our interest to fine-grained entity typing that has recently attracted increasing attention.

Rule-based NER
Early attempts to build NER systems mainly rely on hand-written grammar and rules designed by domain experts for NER challenges, e.g., MUC-7 [19]. The context of named entities is critical for rule-based approaches for it characterizing the mention of an entity. For example, one rule can be that any single word following "Mr" should be tagged as "PER". Such rules encode information such as graphical information (capitalization, punctuation), syntactic (Part-of-speech tag, syntactic structure) or even semantics (word meanings) related to the current word itself as well as its context.
1 Synsets (country, person etc.) in WordNet are used as the reference of named entity types [20]. All the words in the same synset are sent as queries to a search engine to retrieve relevant documents. They then collected words appearing in the context of the query words and calculated the frequency. For an input word along with its context, they compared the context words with the contexts of synsets. The word will be assigned with the corresponding named entity type of the synset whose context is most similar to that of the given word.
Evans et al. [21] built a document-specific typology for named entities using manually designed patterns, e.g., "Y such as X" or "Y like X". Then, they performed clustering on the set of hypernyms in order to identify potential named entity types.
Afterwards, a classification was trained with the typology.
Rule-based approaches rely on hand-crafted rules that are expensive to obtain and require additional efforts when adapted to new data or domain. In addition, rule-based approaches are also vulnerable to noises and variations, even though some of the rules can be encoded as Finite-State-Transducer which increases its capability of being generalized.

Learning-based NER Model
Rather than using manual rules encoding the knowledge of experts, machine learning approaches derive knowledge in a data-driven manner and have been widely used for the task of NER. Machine learning approaches typically model the two subtasks of NER as a sequence prediction task using machine learning based approach: given a word sequence W = w 0 w 1 w 2 . . . w n , the objective of NER is to find the optimal tag sequence T = t 0 t 1 t 2 . . . t n , where the tag set is pre-defined. One of the most popular tagging schema used by NER approaches is the BIO encoding. The set of BIO tags includes Beginning, Inside, and Outside tags. For example, the BIO scheme for person names includes B P ers , I P ers and O, which indicate the beginning of a person name, an internal word of a person name and the word belonging to none of the named entity types. Table 2.1 shows a more specific example of MUC7 where the start and end of Martin Puris is indicated by "B-PER" and "I-PER" respectively. Given the tagging schema and word sequence, the sequence labeling model searches for the best tag sequence with the parameters learned from training process. In the following, we review the popular sequence labeling models having been used for NER.  Charniak et al. [22] proposes a Hidden Markov Model(HMM) for Part-of-speech tagging. Given the word sequence W = w 0 w 1 w 2 . . . w n , the best tagging sequence HMM model assumes that the current word w i is generated from a multinomial distribution conditioned on current tag while current tag is only conditioned on previous tags. In a first order HMM, the above joint probability can be expanded to However, the assumption made by HMM is too strong for NER as the current tag depends on not only the previous tags but also the other types of characteristics in context. For example, in HMM models are not flexible enough to train with a rich feature set as the dependency between words and tags are strictly constrained. In order to overcome this shortcoming, Maximum Entropy (ME) model is proposed as an alternative to HMM.
It has been widely applied to tagging tasks such as POS tagging for it has the freedom of using arbitrary features.
Similar to HMM-based NER, ME-based model assigns each word in the input with either a tag corresponding to the named entity type and indicating if it is the beginning, or a tag indicating it is not part of an named entity.
Given the word sequence W 1...n , the probability of tag sequence T 1..n is calculated as: where Z(T 1...n ) is the partition function defined as: The objective of learning process of ME model is to find the optimal setting of parameters alpha j , and during the inference phase, a Viterbi decoding algorithm can be used for finding the most likely tagging sequence for the given input.
One problem with ME model is known as the label bias problem. When there is no alternative transition from one certain state to another, the model will ignore the observation. In other words, when estimating P (t i |t i−1 , w i ), the observed word w i will not have any influence on the model if t i is the only outgoing transition from t i−1 . As an alternative to ME model, Conditional Random Field (CRF) [3] is proposed in order to avoid such label bias problem by estimating P (t i |t i−1 , w i ) based on the complete sequence of tags. CRF shares the same exponential probabilistic form as ME model except that CRF model is an undirected graph model. Both HMM and ME are directed models while CRF uses an undirected structure which allows the dependency between words and tags to go in both directions. However, long-range dependency becomes intractable as the size of possible features grows exponentially with the number of words in the sequence. To significantly reduce the possible searching space, a simplified version of CRF models allowing only dependency of tags between consecutive time slots, i.e., t i−1 and t i , which is called linear-chain CRF (as shown in As defined by Lafferty et al. [3], linear-chain CRF defines the conditional probability of T given the word sequence W : while Z is the partition function , where T is the tag sequence and W is the word sequence; α j is the weight associated with the j th feature; and phi j is the j th feature function. Based on the above definition, one of the advantages of CRF is that it dynamically models the dependency between possible tags. In addition, it allows us to define arbitrary feature functions for T and W . If the word appears as uppercase more frequently than lower case Type of the initial word, i.e., uppercase, lower case, digit, etc. Word shape, i.e., each character in the word is mapped to its type It can be found that some of the manually designed features shares the same intuition of the hand-crafted rules used by a rule-based system. Basically, the feature set consists of grapheme features of the word at the current step as well as previous and next steps as contextual markers of the entities.
The manually designed features require human knowledge and are hard to be generalized to new domains or new datasets. In comparison, the second class of features are data-driven and can be automatically learned without the need for intervention.
Most notably, NER models [23,4,25] using pre-trained word embeddings (or phrase embedding) as features have successfully achieved state-of-the-art performance. Word embedding is a type of vector representation of words that is typically learned via auxiliary tasks such as language modeling [27]. We will cover more reviews on the technique of learning embeddings of words or concepts in section2.5.

Fine-grained NER
Fine-grained named entity recognition refers to the task of recognizing named entities belonging to a fine-grained set of entity categories. As compared with coarse-grained NER, the fine-grained entity types are organized into a hierarchy. In contrast to coarse-grained NER, the fine-grained NER is generally modeled in two separate steps: 1) segmentation, which is to detect the boundaries of named entity mentions; and 2) classification, which is to classify the detected named entity into fine-grained types. It is also likely that the classifier might have accesses to pre-existing entity boundaries rather than reading the outputs of the detection model. In such case, classification of 8 entity mentions becomes the only focus of Fine-grained NER, which is also known as the task of Fine-grained Entity Typing (FNET).
One of the biggest challenges faced by fine-grained named entity recognition is the shortage of training data. Since the number of fine-grained types is much bigger than coarse-grained of named entities, it is harder to have sufficient training data for each category. Fortunately, structural information in public resources such as Wikipedia and Freebase can be used for automatically building annotation of fine-grained named entity types. In this section, we review the most recent work on FINER or FNET: FIGER [18], HYENA [28], GFT [5], the embedding based methodyogatama:2015, as well as methods based on deep neural networks [7,29].

Fine-grained Labels
The first step to creating a benchmark for fine-grained NER is identifying the set of labels (or classes) of named entities. Different from coarse-grained NER, whose label set is typically limited to a few categories such as Organization, Location, and Person, there's actually no standards for fine-grained types. However, previous works have used online knowledge base for building the label set as well as their hierarchy.
FIGER derives a set of 112 named entity types from Freebase, an online knowledge base built on Wikipedia and WordNet, even though the hierarchy only has two levels.
HYENA uses a much larger tagset composed of 505 entity types by selecting top 100 subtypes for each top-level types of YAGO knowledge base [30,31]. Additionally, the labels used in their approach are organized as a taxonomic hierarchy. More recently, GFT manually refined and expanded the label hierarchy used by FIGER into three levels. As shown in Fig. 2

Automatic Generation of Training Data
Training classifiers for fine-grained entity types requires a considerable number of labeled data, which is expensive and time-consuming. Instead, previous methods 9 Figure 2.3: Hierarchical categories used in Ling et al. [5] have been trying to use online resources such as Wikipedia to automatically construct Instead of using the Wikipedia categories, Ling et al. [18] propose to use the freebase categories which contains fewer noises. By creating the mappings from the classic form to a freebase category and from freebase to manually refined labels, a large annotated corpus (approx. 3 million sentences) is built from the anchor texts of Wikipedia pages and Freebase. However, the labels in the annotated data built by [18] still contain noises, because the mapping from classic forms to freebase concepts do not consider the impacts of context during tagging. For example, ishington can be a military leader or a city according to Freebase, but in FIGER data, no matter what the contexts is, W ashington is always annotated with both the two labels. To overcome this shortcoming, [5] used the context of named entities to refine the labels with the help of a set of heuristic rules which takes into account the contexts of named entities: Sibling, using the parent label, if the mention is tagged with multiple labels having the same parents; Coarse-type, using softmax classifiers trained on ACE corpus, which classify named entity mentions into top-level labels based on their local context, and removing finegrained labels that are inconsistent with the results of coarse-grained classification; Minimum count, removing the labels appearing less than a certain number of times within the current documents. These heuristic rules actually result in training instances with higher confidence as compared to FIGER.

Features
Fine-grained NER methods convert the input into a vector of features. Most of the features used in fine-grained NER are shared across existing methods. In Table 2.2, we list the features have been used in all of the four systems. We observed that most of the features are adapted from coarse-grained NER.

Feature Description Example Phrase head
Head word of a named entity mention "Obama" Non-head All the non-head words of the mention "Barack", "H" Head cluster Word cluster ID of the head word "59" Character n-gram character n-grams of the head word ":ob","oba",...

Word Shape
The word shape of the mention "Aa A Aa" Role Dependency label of the head word "nsubj" Context context word of the mention "B:who", "A:first" Parent Lexical parent in a dependency tree "picker"

Segmentation and Multi-label Classification
Depending on the settings of experiments, different methods might use different strategies to obtain the segmentation of named entity mentions. For example, human-annotated segmentation of testing data might be directly used as input [5,6,7,29], even though the assumption that the entity boundaries are pre-existing sometimes does not hold in reality. HYENA uses the segmentation provided by the anchored text of Wikipedia page that is not as reliable as manual annotations. In contrast, the setting of FIGER is the most reasonable and practical one. Their testing set is manually labeled. They train a CRF model with BIO tagging schema for segmentation of text into named entity mentions and non-mention during testing.
The classification of fine-grained NER is typically modeled as a multi-label classification problem. According to Silla et al. [32], most of the existing multi-label classifiers fall into four categories: 1) flat, referring to those using a single multi-class classifier; 2) local, referring to methods using one-versus-rest binary classifiers for each node in the hierarchy; 3) local per parent node, associating a multi-class classifier with each internal node; and 4) global, using a single multi-class classifier with objective function encoding the similarity between labels. As an example of the first category (flat), [18] adapted a simple linear classifier based on single perceptron [33], which maps an input x to a label y given the feature function F (x, y) = [f 1 (x, y), . . . , f n (x, y)], where w = [w 1 , . . . , w n ] are the vector of weights learned from training set. During training, w is updated by where {y} are the set of true predictions, and {ŷ} are the set of mispredictions, which is discouraged by decreasing the weights associated with the corresponding features.
In order to get multiple labels for one input x, all the tags y with positive score are selected as the predicted labels. All the above three methods, i.e., FIGER, HYNNA, and GFT, do not consider the similarity between labels, which would be very useful for handling the data sparseness when a big label set is in use. Instead of using traditional classification methods, WsabieNER used an embedding-based method, called Wsabie [34], which maps both labels and features to same lower dimensional space using liner transformations is the vector of features, and y is the vector of labels. The classification is done by scoring a pair (x, Y ) with the products of the transformed vectors, To learn the parameters, A and B, from training data, they optimize an objective function called weighted approximate pairwise (WARP) [34], which is designed to rank the true prediction y higher than the mispredictionŷ. The mathematical description where rank(x, y) is margin infused counting function defined as , and R(k) = k i=1 1 i is a function turning the rank into a weight. By optimizing this objective function, F (x) and y are mapped to lower dimensional space where labels cooccurring with similar feature vector F (x) are close to each other. Fig.2.5 shows that in 2-dimension space (after applying PCA, and labels are colored based on their top-level labels), labels sharing parent node or superclass tend to be grouped together.
As the prediction is made based on the products of label and feature embeddings, this also suggests that the classifier is less discriminative to labels of a lower level, which tends to be close in new space, while more discriminative to labels of a higher level.

Label Noise Reduction
As mentioned above, data used for training FNET models are weakly tagged. It is unavoidable that the weak tagging process would introduce label noises. More specifically, the noisy instances are mostly over-labeled because of the heuristic rule of weak tagging. For example, in the dataset created by Ling et al. [18], each instance is assigned with all the possible entity types if the surface form matches with multiple canonical forms. Some previous studies [35] have formulated this problem as a partial label learning problem where the true label set is a subset of the observed label.  label set. For example, one of the loss function used in the previous work [35] has the form like where Y refers to the set of negative labels. The intuitive behind this loss function is to learn from only the most confident label of a noisily labeled instance.

Neural Architecture for FNET
Following the trend in adopting deep neural networks for NLP tasks, recent work on FNET has adapted the LSTM to modeling the sequence of contextual words as well as of entity mentions. For example, Shimaoka et al. [7](as illustrated in Fig. 2. 6) proposes to encode the contextual word sequences using bi-directional LSTM. Their framework also incorporates an attention component to model the salience of each contextual word in resolving the types of the center entity mention. A similar neural network is also adopted by work of Abhishek et al. [29] which replaces the bag-of-words encoder of entity mentions with another uni-directional LSTM encoder. Their work shows that the neural architecture is able to perform comparably than embedding methods using the manually designed features.
One question is how to encode the correlation of labels. In other words, the labels are not created free of semantics. Each label in the hierarchy, in fact, corresponds to a concept in real world. It is, therefore, useful to model the correlation or dependency of labels. In HYENA [28], although the classification results are passed from parent to its children, each classifier, however, is trained independently. In comparison, the embedding based methods [6,36] captured the similarity between labels by considering the concurrence of labels and features. But, their methods fail to take into consideration the existing hierarchy that has encoded useful information about the dependency between labels. To address these issues of existing approach, one solution is to consider the Bayesian hierarchical classifier [37], which models the dependency between labels using a hierarchical prior. Their experiments have shown improved performance over f lat and local classifiers on a range of tasks and datasets and is scalable to work with large-scale training data.

Targeted Sentiment Analysis
In this section, we survey multiple research areas related to the proposed framework, namely: Aspect-based Sentiment Analysis (ABSA), targeted sentiment analysis, targeted ABSA, and finally works on incorporating external knowledge into deep neural models.

Aspect-Based Sentiment Analysis
ABSA is the task of classifying sentiment polarity with respect to a set of aspects.
The biggest challenge faced by ABSA is how to effectively represent the aspect-specific sentiment information of the whole sentence. Early works on ABSA have mainly relied on feature-engineering to characterize sentences [38,39]. Motivated by the success of deep learning in representation learning, many recent works [40,41,42,43] utilize deep neural networks to generate sentence embeddings (dense vector representation of sentences) which are then fed to a classifier as a low-dimensional feature vector.
Moreover, the representation can be enhanced by using the attention mechanism [43], which is typically a multi-layer neural network taking as input the word sequence and aspects. For each word of the sentence, the attention vector quantifies its sentiment salience as well as the relevance to the given aspect. The resulting sentiment representation benefits from the attention mechanism for it overcomes the shortcoming of recurrent neural networks (RNNs), which suffer from information loss when only one single output (e.g., the output at the end of the sequence) is used by the classifier.

Targeted Sentiment Analysis
Targeted sentiment analysis aims to analyze sentiment with respect to targeted entities in the sentence. It is thus critical for targeted sentiment analysis methods, e.g., the target-dependent LSTM (TDLSTM) and target connection LSTM (TCLSTM) [44], to model the interaction between sentiment targets and the whole sentence. In order to obtain the target-dependent sentence representation, TDLSTM directly uses the hidden outputs of a bidirectional-LSTM sentence encoders panning the target mentions, while TCLSTM extends TDLSTM by concatenating each input word vector with a target vector. Similar to ABSA, attention models are also applicable to targeted sentiment analysis. Rather than using a single level of attention, deep memory networks [45] and recurrent attention models [46] have achieved superior performance by learning a deep attention over the single-level attention, as multiple passes (or hops) over the input sequence could refine the attended words again and again to find the most important words. All existing approaches have either ignored the problem of multiple target instances (or words) or simply used an averaging vector over target expressions [45,47]. Unlike such approaches, our method weights each target word with an attention weight so that a given target is represented by its most informative components.

Targeted Aspect-Based Sentiment Analysis
Two baseline systems [48] are proposed together with SentiHood: a feature-based logistic regression model and an LSTM-based model. The feature-based logistic regression model uses feature templates including n-grams tokens and POS tags extracted from the context of instances. The LSTM baseline can be seen as an adaptation of TDLSTM that simply uses the hidden outputs at the position of target instances assuming that all target instances are equally important.

Incorporating External Knowledge
External knowledge bases have been typically used as a source of features [49,50,51]. Most recently, neural sequential models [52,53] leverage the lower-dimensional continuous representation of knowledge concepts as additional inputs. However, these approaches have treated the computation of neural sequential models as a black-box without tight integration of knowledge and computational structure. The proposed model, termed Sentic LSTM, is inspired by [54], which adds a knowledge recall gate to the cell state of LSTM. However, our method differs from [54] in the way of using external knowledge to generate the hidden outputs and controlling the information flow.

ASR Hypoehses Reranking
Another research line to be reviewed is ASR hypotheses reranking. It is a technique to improve the quality of texts automatically transcribed by a speech recognition system. ASR hypoehteses reranking is critical for understanding opinions expressed via not only texts but also audios. ASR hypotheses reranking might sometimes be referred to as discriminative language modeling in contrast to generative models such as the n-gram model used by [55].
Typically, ASR systems consist of two modules:1) acoustic model and 2) language model. The best hypothesis of an ASR system is induced with where A is the acoustic representation of an utterance, and W is the hypothesized word sequence, respectively. P a (s|a) is the acoustic model, and P l (s) the language model with α controlling the relative importance of the language model. Speech transcripts contain disfluencies such as repetitions, false starts and fillers, which can interrupt the consecutive n-gram patterns. The conventional n-gram language model, therefore, is unsatisfactory for modelling spoken language, especially for conversational speech which is spontaneously incurred. In order to address the problems caused by the informal nature of speech, some works have modeled the long distance context using syntactic parse trees. Approaches using syntactic information for language modeling either incorporate the parsing process into language modeling or maintain a separate process by including features generated from syntactic parsing. As an example of the first type of approaches, [56] used a shift-reduce parser to maintain a set of possible parses of a given sentence. The probability of a word given its previous context is then defined as the sum of probabilities of candidate parses. As for the second type, [57,58] used single perceptron model with features extracted from syntactic parse trees and dependency tree. Their experiments show that, when combined with the ASR baseline model, discriminative reranking could further reduce the word error rate, even though by a small margin. A feature function F (W ) is defined for a given word sequence W, and used for rescoring the hypothesis of ASR with a modified log probability log(P (W, A)) = log(P a (A|W )) + α log(P l (W )) + where f i is the i th feature in F (W ), and w i is its weight. The syntactic features used by [57] are derived from the parse trees produced by a syntactic parser, including head features such as the syntactic head of a detected noun phrase, and head-tohead dependencies. Instead of using syntactic parsing based features, [58] extracted features from dependency parses, which are mainly composed of a set of dependency tuples. For example, for a sentence

I WENT TO THE AN-THE DOCTER
Stanford parser produces the set of tuples as shown in Table.2.3. Each tuple represents a dependency link in the parsing result, including two nodes and their syntactic relation such as noun subject(nsubj ) between a verb and subject. All the tuples generated for the given sentence are added to the feature set.
It is worth a further investigation on whether reranking can benefit from reducing the data sparseness.

Embedding Models
As discussed in previous sections, many NLP models transform the input into feature vectors of very high dimensionality. Computation on such high-dimensional space is very expensive and sometimes even intractable. To address this issue, dimensionality reduction tricks have been adopted to convert the original feature vector into lower dimensional representations which are also known as embeddings. As compared with one-hot representation whose dimensionality is typically same as the size of feature dictionary, embedding approaches could significantly reduce the dimensionality without suffering a significant loss of information. Depending on the tasks, embeddings can be generated for not only words but also concepts or phrases.
One of the fundamental intuition of learning embeddings for words or concepts is distributional semantics which assumes the distribution of contexts characterizes the semantic meaning of a central word. In other words, two words (or concepts) are close in semantics if having the similar distribution of contexts. Recently, embedding based on distributional semantics have been used as effective features for many NLP tasks, 20 e.g., named entity recognition [4], dependency parsing [59], semantic relation classification [60], antonym detection [61], sentiment analysis [62], and spoken language understanding [63].
According to the classification by Baroni et al. [64] and Levy et al. [65], existing word embedding methods based on distributional semantics can be grouped into two categories: count-based and prediction-based. The count-based methods can be dated back to methods such as Latent Semantic Analysis (or Latent Semantic Indexing) [66] that are followed by lines of research on decomposing matrices of co-occurrence counts.

Count-based Models
It is a long history that the co-occurrence counts and reweighting tricks have been used for obtaining embeddings. Here, we are only able to review a small subset of them that have recently attracted enough attention.
First of all, the task of learning embeddings from co-occurrence counts can be formulated as a matrix factorization problem. Let M ij be an entry of M . Given the training dataset D = {w 1 , w 2 , · · · , w n }, the value of M ij can be either original or weighted counts of co-occurrence of word w i and a context c j . For example, a popular weighting schema is Pointwise Mutual Information (PMI) [67].
We then formulate the process of deriving word embeddings as a mapping from co-occurrence matrix M ∈ R n×m to embedding matrix W n×d , with each row of W corresponding to a d-dimensional vector representation of the word w i .
Next, we will review two of the count-based methods that are recently studied for learning embeddings of words or concepts.

Single Value Decomposition
A natural choice to decompose the co-occurrence matrix to lower-dimension vectors is to use traditional dimension reduction methods such as Single Value Decomposition (SVD).
SVD generates an optimal low-rank approximation of the original counts matrix M by factorizing it as where U ∈ R n×n and V ∈ R m×m are unitary matrices, and Σ ∈ R n×M being a diagonal matrix. Each diagonal entry of Σ is a singular value of M . To get a d-dimensional vector representation, we just take the dot product of U * and Σ * , i.e., where Σ * is the same matrix as Σ except that it contains only the top d values, and U * being the corresponding truncated matrix of U .

Random Projection
One problem with SVD is that it is hard to scale up, especially when the original matrix is sparse and too high-dimensional [68].
Cambria et al. [10] propose to a data oblivious method, called random projection, to map the original high-dimensional space to a lower dimensional space. The adopted random projection of concept into the new space is done via a Gaussian matrix.
According to Johnson and Lindenstrauss (JL) Lemma [69], points distributed in a m-dimensional space can be safely projected to a d-dimensional space, where d is logarithmic to m, and pairwise distances being preserved with high probability. It is, therefore, possible to use a random matrix to transform each row of original cooccurrence matrix to a lower-dimensional row vector.
Cambria et al. [10] propose to use a fast random projection technique, called Subsampled Randomized Hadamard Transform (SRHT) [70] a random matrix Φ, which can be factorized by where R is a d × m with rows being sampled from an uniform distribution, and H is a m × m normalized Walsh-Hadamard matrix, and D is m × m random matrix.
After computing Φ, the new embedding matrix W can be obtained by linearly transforming M using the transpose of Φ, i.e., As shown by Cambria et al. [10], the random projection method could project the very high-dimensional concept matrix to a low-dimensional space with high efficiency.

Prediction-based Models
Apart from working with co-occurrence counts, the past few years have seen a trend of learning embedding by maximizing the probability of contextual terms. The family of embedding learning methods [23,71,72] are, therefore, classified as prediction-based methods. Next, we will review one of the most widely-used prediction-based methods, called Skip-gram model, as well as a variant of it, called CBOW model.

Skip-gram Models
Skip-gram model [72] is a neural network model that can be efficiently learned from a large-scale corpus with billions of words. Figure Its objective function is the sum of log probabilities p(w i+j |w i ) over the whole corpus, where w i+j indicates a neighbor word of w i , k the size of the context window and N is the length of the dataset. And, the basic Skip-gram formulation defines p(w i+j |w i ) using the softmax function as follows: where W is the vocabulary of words, v w i+j and v w i are the context and word embeddings, respectively.

Continuous Bag-of-Words (CBOW) Model
The training objective CBOW model [72] is to maximize the sum of probability p(w i |w i−k , . . . , w i+k ), which is similar to Skip-gram model.
Cross range leads to delta wing .
center word context window leads --> cross leads --> range leads --> to leads --> delta Cross range leads to delta wing .

Summary
In this chapter, we have reviewed recent progress in several tasks that are core to understanding human opinions. We also survey on techniques aiming to learn embedding for words and concepts. All existing solutions have their own technical limits as well as unsolved issues. However, it lacks investigations on leveraging the concepts for specific tasks. In addition, the fusion of information from concept-level and other levels (e.g., word-level) has been missed in existing studies.

Chapter 3
Learning Word Embedding with Multi-task Entity-based Features

Introduction
Named entity recognition (NER) has been studied and applied successfully to formal [73] and informal texts [74] for many years. Nevertheless the adaptation of NER methods to conversational speech remains challenging due to, for example, case insensitivity, lack of punctuations, un-grammatical structure, repetition, and presence of disfluencies inherent to conversations. In addition, there is not much spoken data annotated with named entities to cover the huge variety of named entity instances likely occurring in speech, and simply increasing the amount of manual annotation is not realistic for reasons of cost, evolution of new spoken terms and diversity.
Several works on NER from spoken contents have already explored the use of external resources like online gazetteers [75] and Wikipedia [76] to overcome the lack of annotations. Gazetteers, for instance, have successfully boosted NER performance for given entities (e.g., Location), but do not convey the information related to the context words surrounding the entity names that are also important for NER. Other lexical resources such as WordNet provide semantic relations like synonymy among common English words, but remain limited for names. A second category of approaches [77,78,79,80,81] tackling the sparseness of NER training data use unlabeled data to learn low dimensional vector representation of words, called word embeddings. Used either as continuous [77,78,81] or discrete features [79,80], word embeddings have been shown effective in improving the generalization of NER.
For the last two years, an increasing number of studies have suggested that injecting application-specific information into the neural networks used to train word embeddings can further improve the performance of down-stream applications, e.g., dependency parsing [59], semantic relation classification [60], antonym detection [61], spoken language understanding [63]. The task-specific information used by these methods are injected into the training process of word embeddings mainly by expanding or replacing the input or output of the neural network. Passos et al. [82], to our knowledge, is the only work trying to learn word embeddings for NER by leveraging lexicons related to named entities. Our proposal is inspired by their approach and can be seen as its generalization in the sense that our word embedding is trained with multiple relevant tasks while their model is trained on only one additional task that predicts whether the current word belonging to a predefined set of semantic classes.
The key to learning word embeddings for the particular problem of NER is to encode entity-related information into the word embedding space. As the first step to investigate a tight fusion of concept-level 1 and word-level information, we propose to augment the learning process of word embedding with entity-related features. More specifically, we introduce an NER framework including a modified word embedding method based on Skip-gram models [72], in which we generalize the training objective to integrate features explicitly designed for NER task and grouped by types. From a different point of view, we extend the training of Skip-gram model to a multi-tasks setting where each feature type corresponds a particular task. The multi-task learning of word embeddings allow the embedding to form a thorough representation of the word by incorporating information from different information sources. We prove the effectiveness of our method in the context of NER. Namely, we find that injecting such NER-specific information as part-of-speech tags, taxonomic relations, and self-training features [76] yields improvements over baseline (i.e., without using word embeddings), when evaluated against conversational speech transcripts (i.e., Switchboard Corpus).
Furthermore, the results show that our proposed word embeddings also outperform the task-independent word embeddings.

Methods
The proposed NER framework illustrated in Figure 3.1 is composed of two parts: 1) a task-specific word embedding learning integrating features specific to NER (Part I in

Baseline NER method
In this section, we describe the baseline NER method, a conventional linear-chain CRF with BIO encodings. The features extracted from every word w i of the training data and used to train the CRF are summarized in Table 3.1. The feature set consists of ngrams, part-of-speech (POS) tags, affixes, and the BIO tag of the i th word (designated as y i ).

Context Features (CF )
Unigram  where k is size of context window which are a certain number of words surrounding the central word. The probability p(c|w) is given as c exp(v c v w ) However, as computing the gradient of log p(c|w) requires iteration over all pairs of (w, c), optimizing this objective function is intractable when the size of context vocabulary is large. To approach this problem, [72] uses negative sampling as an alternative training objective. The objective function can be rewritten as where σ(x) = 1/(1 + exp(−x)). For each (w, c) pair appearing in the corpus, N pairs of (w, c k ) are randomly sampled from a unigram distribution P w as negative training instances.

Feature extraction for word embeddings
First of all, we extract four types of features (i.e., neighboring words, part of speech (POS), taxonomic, self-trained) from the unlabeled data to train word embeddings.
In principal, they are mostly features used for training the baseline NER model, and one of the intuition of our model is to learn a condensed version of these features from unlabelled data to achieve higher capability of generalization.
Neighboring words and POS tags: They are acknowledged to be efficient for NER [75,76]. In fact, we use these features not only for training our word embeddings, but also for training the baseline model of NER (Section 3.2.1). Their formal definitions can be found in the first 4 lines of Table 3.1. In terms of feature type, we define two separate types for words and POS tags respectively.
Taxonomic features: The generalization of similar entities within a common category (or concept) in a taxonomy may be useful for NER to capture contexts shared by similar entities. We select 280 concepts with at least 100 instances from ConceptNet, which is a large, automatically created semantic graph including taxonomic relations. For each concept g, we add binary taxonomic features, which are defined as TX g (w i+k ), 2 ≤ k ≤ 2, where i is the index of current word, and TX g (w) indicates if the word w belongs to the concept g or not.
Self-trained features: As in [83], these features are generated by automatically labeling the training data using a baseline NER tagger (Section 3.2.1). We use the resultant named entities hypotheses as additional features for word embeddings. The features are defined as T i+k , with T j the NER label of the j th word.

Learning word embeddings from entity-based features
Our proposal is based-on the Skip-gram model [72], a neural language model that can be efficiently trained on a large corpus of billions of words. Its objective function is the sum of log probabilities p(w i+j |w i ) over the whole corpus, where w i+j indicates a neighbor word of w i , k the size of the context window and N is the length of the dataset. And, the basic Skip-gram formulation defines p(w i+j |w i ) using the softmax function as follows: where W is the vocabulary of words, v w i+j and v w i are the embeddings of context word and current word, respectively. We modify the Skip-gram model to predict the set of features F (w i ) extracted for the given word w i at the i th position of a corpus.
The objective function can be rewritten: To estimate p(w i+j |w i ), the basic Skip-gram model assumes a single distribution over all the words. However, for p(f |w i ), since the features we use for training application-specific word embeddings are heterogeneous, they may have different distributions. We thus split the whole set of features (S) into subsets (S X ), where X indicates one of the four feature types aforementioned and the relative position to the center word. For example, the subset S pos:−1 includes all the features that tell the POS tags of the previous word. We define C S (f ) as the function that returns the subset of S that contains the feature f . We define the probability of extracting f for a given word w from the training data as follows: where v w and v f are the vectors associated with center word w and feature f respectively, and both are the parameters to be learned. Note that, with the new definition, the objective function becomes a linear combination of the objective functions of multiple classifiers with equal weights. We optimize this objective function using stochastic gradient descent and negative sampling method previously proposed for Skip-gram [72], which rewrites the objective function as: In this expression σ(x) = 1/(1 + exp(−x)), and the set of features Z(f ) is created by randomly selecting n negative samples from a unigram distribution over features in

Multi-task interpretation of the proposed method
Since each type of features are heterogeneous to one another, we could also view the prediction of each feature type as performing a separate task. Therefore, the iteration over the set of feature type (i.e., F ) can be seen as performing multi-task learning over a set of tasks. For example, predicting neighboring words and POS tags corresponds to the tasks of language modeling and POS tagging respectively.

Using word embeddings as NER features
We convert word embeddings to additional features of the CRF model as in [79,80] by binarizing vector elements of word embeddings, and clustering of words based on similarity of word embeddings. The vectors are binarized using the following rules, The features learned from word embeddings are summarized in Table 3.2. We compare the proposed Skip-gram model (Skip NER ) with non-task specific word embeddings models previously used for NER. We retrained the word embeddings for Skip-gram, CBOW [72], and Glove [81] with the ukWaC corpus, using two-word context windows for each side of the current word. We used the pre-trained word embeddings models 3 of HUANG [71] and SENNA [77], because their training is slow.

Evaluations
We report in  Table 3.5: Performance of NER methods in terms of F-score In the second experiment, we investigate the impact of the amount of labeled training data used to train the NER model. As illustrated in Figure 3.2, the Skip NERbased NER system always outperforms the other systems, independent of the amount of training data. Also, the NER systems with the generic (or non task-specific) word embeddings also perform better than the baseline. It suggests that the generalization brought by word embeddings consistently helps NER models in dealing with data sparsity. As more training data are used (80%-100% of training data), the gap of performance between all the baseline systems reduces. However, the gap between the baseline and Skip NER further increases even using more training data, which may mean that our method robustly handles data sparsity and extracts more discriminative information when more data are available.
We also study the impact of each subset of features used to train the Skip NER model. As reported in Table 3.6, we find that using the whole set of features outper-  These observations suggest Skip NER is able to capture more discriminative information for NER compared to generic word embeddings.

Summary
In this work, we propose a novel method to train task-specific word embeddings for NER by incorporating NER related multi-task learning. Through several experiments, we have shown that, on the manual transcripts of open-domain conversations, how our proposed feature-enriched word embeddings can out-perform baseline NER method and systems using task inde-pendent word embeddings.

35
Chapter 4 Label Embedding for Fine-grained Named Entity Typing

Introduction
Named entity typing (NET) is the task of inferring types of named entity mentions in text. NET is a useful pre-processing step for many NLP tasks, e.g., autocategorization and sentiment analysis. Named entity linking, for instance, can use NET to refine entity candidates of a given mention [18]. Besides, NET is capable of supporting applications based on a deeper understanding of natural language, e.g., knowledge completion [86] and question answering [87,88]. Standard NET approaches or sometime known as named entity recognition [11,13,89] are concerned with coarsegrained types (e.g., person, location, organization) that are flat in structure. In comparison, fine-grained named entity typing (FNET) [18], which has been studied as an extension of standard NET task, uses a tree-structured taxonomy including not only coarse-grained types but also fine-grained types of named entities. For instance, given "[Intel ] said that over the past decade", standard NET only classifies Intel as organization, whereas FNET further classifies it as organization/corporation.
FNET is faced with two major challenges: growing type set and label noises. Since the type hierarchy of entities is typically built from knowledge bases such as DBpedia, which is regularly updated with new types (especially fine-grained types) and entities, it is natural to assume that the type hierarchy is growing rather than fixed over time.
However, current FNET systems are impeded from handling a growing type set for that information learned from training set cannot be transferred to unseen types.
Another problem with FNET is that the weakly supervised tagging process used for automatically generating labeled data inevitably introduces label noises. Current solutions rely on heuristic rules [90] or embedding method [35] to remove noises prior to training the multi-label classifier. In order to address these two problems at the same time, we propose a simple yet effective method for learning prototype-driven label embeddings that works for both seen and unseen types and is robust to the label noises. The label embeddings, which might also deemed as concept embeddings, are inferred from a pre-trained word embedding space via a set of prototypes. From the perspective of fusing information from multiple levels, we made an attempt to fuse word-level information (i.e., word embedding of prototypes) to represent the canonical form of concepts (i.e., concept types). Another contribution of this work is that we combine prototypical and hierarchical information for learning label embeddings.
The remainder of this chapter is organized as follows: Section 4.2 proposes a survey of prior works related to FNET; Section 4.3 introduces the embedding-based FNET method and its zero-shot extension; Section 4.4 describes our label embedding method; Section 4.5 illustrates experiments and analysis for both few-shot and zeroshot settings; finally, Section 4.6 concludes the paper and discusses future work.

Background
There is little related work specifically on zero-shot FNET but several research lines are considered related to this work: fine-grained named entity recognition, prototypedriven learning, and multi-label classification models based on embeddings. As FNET works with a much larger type set as compared with standard NET, it becomes difficult to have sufficient training data for every entity type if relying on only manual annotation. Fortunately, semi-structural data such as Wikipedia pages [18] turns out to be a good source for automatically generating training data.This auto-annotating practice has been followed by later work on FNET [28,6,35]. However, since the automated tagging process is not perfect all the time, a number of label noises are propagated to supervised training and have consequently a negative impact on the classification performance.
The starting point of this work is the embedding method, WSABIE [34], which have been adapted by [6] to FNET. WSABIE maps input features and labels to a joint space, where information is shared among correlated labels. However, the joint embedding method stills suffers from label noises. In addition, WSABIE fails to handle unseen types for it uses labeled training set as the only source learning label embeddings.
In addition to WSABIE, this work is also inspired by recent progress on image annotation based on embedding methdos. For example, DeViSE [91] is proposed for annotating image with words or phrases, where embedding of single words, e.g., fruit, that that are pre-learned from textual data using Skip-gram word embeddings (see Section 2.5.2.1 for more details) [72] are directly used for representing labels. In addition to label itself, existing work has also attempted to derive label embeddings from side information such as attributes [92], manually-written descriptions [93], taxonomy of types [94,92,95], and so on and so forth.
Another related line of research is prototype-driven learning. In [96], a sequence labeling model uses prototypes as features and is applied to NLP tasks such as partof-speech (POS) tagging. Prototype-based features [80] are then adapted for coarsegrained named entity recognition task. Even though we select prototypes in the same way as [80], we use prototypes in a very different manner: we consider prototypes as the basis for representing labels, whereas prototypes are mainly used as additional features in prior works [96,80]. In other words, prototypes are previously used on the input side, while we use them on the label side.

Embedding Methods for FNET
In this section, we introduce the embedding method for FNET proposed by [6] and its extension to zero-shot entity typing.

Joint Embedding Model
Each entity mention m is represented as a feature vector x ∈ R V ; and each label y ∈ Y is a one-hot vector, where Y is the set of true labels associated with x.Ȳ denotes the set of false labels of the given entity mention. The bi-linear scoring function for a given pair of x and y is defined as follows:

Zero-shot FNET Extension
A zero-shot extension of above WSABIE method can be done by introducing pretrained label embeddings into the framework. The pre-trained label embeddings are learned from additional resources, e.g., text corpora, to encode semantic relation and dependency between labels. Similar to [92], we use two different methods for incorporating pre-trained label embeddings. The first one is to fully trust pre-trained label embeddings. Namely, we fix B as the pre-trainedB and only learn A in an iterative process. The second method is to use pre-trained label embedding as prior knowledge while adjusting both A and B according to the labeled data, i.e., adding a regularizer to the WARP loss function, where || · || F is the Frobenius norm, and λ is the trade-off parameter.

Prototype-driven Label Embedding
Joint embedding methods such as WSABIE learn label embeddings from the whole training set including noisy labeled instances resulting from weak supervision. It is inevitable that the resulting label embeddings are affected by noisy labels and fail to accurately capture the semantic correlation between types. Another issue is that zeroshot frameworks such as DeViSE are not directly applicable to FNET as conceptually complex types, e.g., GPE (Geo-political Entity) cannot be simply mapped to a single natural word or phrase.
To address this issue, we propose a simple yet effective solution which is referred to as prototype-driven label embedding (ProtoLE), and henceforth we useB P to denote the label embedding matrix learned by ProtoLE. The first step is to learn a set of prototypes for each type in the type set. ProtoLE does not fully rely on training data to generate label embeddings. Instead, it selects a subset of entity mentions as the prototypes of each type. These prototypes are less ambiguous and noisy compared to the rest of the full set.
Even though it is already far less labor-intensive to manually select prototypes than annotating entity mentions one by one, we consider an alternative automated process using Normalized Point-wise Mutual Information (NPMI) as the particular criterion for prototype selection. The NPMI between a label and an entity mention is computed as: where NPMI(·, ·) is the point-wise mutual information computed as follows: where p(y), p(m) and p(y, m) are the probability of entity mention m, label y and their joint probability. For each label, NPMI is computed for all the entity mentions and only a list of top k mentions are selected as prototypes. Note that NPMI is not applicable to unseen labels. In such case, it is necessary to combine manual selection and NPMI.
Word embeddings methods such as Skip-gram model [72] are shown capable of learning distributional semantics of words from unlabeled text corpora. To further avoid affected by label noises, we use pre-trained word embeddings as the source to compute prototype-driven label embeddings. For each label y i , we compute its label embedding as the average of pre-trained word embeddings of the head words of prototypes, i.e.,B where v m ik denotes the word embedding of kth word in the prototype list of label y i . In the case of using phrase embeddings, the full strings of multi-word prototypes could be used directly.

Hierarchical Label Embedding
Another side information that is available for generating label embeddings is the label hierarchy. We adapt the Hierarchical Label Embeddings (HLE) [92] to FNET task.
Unlike [92], which uses the WordNet hierarchy, FNET systems typically have direct access to predefined tree hierarchy of type set. We denote the label embedding matrix resulting from label hierarchy asB H . Each row inB H corresponds to a binary label embedding and has a dimension equal to the size of label set. For each label, the sets B H ij to 1 when y j is the parent of y i or i = j, and 0 to the remainder, HLE explicitly encodes the hierarchical dependency between labels by scoring a type y i given m using not only y i but also its parent type P arent(y i ). The underlying intuition is that recognition of a child type should be also based on the recognition of its parent.

Prototype-driven Hierarchical Label Embedding
One shortcoming of HLE is that it is too sparse. A natural solution is combining HLE with ProtoLE, which is denoted as Proto-HLE. SinceB H ∈ R N ×N andB P ∈ R D×N , the combined embedding matrixB HP can be obtained by simply multiplyingB H bỹ Note thatB HP has the same shape asB P , and it is actually representing the child label as a linear combination of the ProtoLE vectors of its parent and itself.

Type Inference
Having computed the scoring function for each label given a feature vector of the mention, we conduct type inference to refine the top k type candidates. In the setting of few-shots FNET, k is typically set to the maximum depth of type hierarchy, while different values for k may be used for a better prediction of unseen labels in zeroshot typing. For top k type candidates, we greedily remove the labels that conflict with others. However, unlike [6], we use a relative threshold t to decide whether the selected type should remain in the final results, which is more consistent with the margin-infused objective function than a global threshold. Namely, a type candidate will be passed to type inference only if the difference of score from the 1-best is less than a threshold.

Experiment Setup
Our method uses feature templates similar to what have been used by state-of-the-art FNET methods [18,90,6,36].

Generating ProtoLE
Our ProtoLE embeddings use Continuous-Bag-of-Words (CBOW) word embedding model [72] trained on Wikipedia dump using a window of 2 words to both directions.
We use 300 dimensions for all embedding methods except HLE.

Few-shots Fine-grained Entity Typing
In this section, we compare performances of FNET methods in the setting of few-shots FNET where the training set covers all types. Methods compared in this section are

Zero-shot Fine-grained Entity Typing
In this section, we evaluate our method's capability recognizing mentions of unseen fine-grained types. We assume that the training set contains only coarse-grained types (i.e., Level-1), and Level-2 types are unseen types to be removed from the training set.

Summary
In this paper, we present a prototype-driven label embedding method for fine-grained named entity typing (FNET). It shows that our method outperforms state-of-theart embedding-based FNET methods in both few-shots and zero-shots settings. It also shows that combining prototype-driven label embeddings and type hierarchy can improve the prediction of coarse-grained types.

Binary Entity Embedding for Reranking ASR Hypotheses
In this chapter, we address the problem of reranking ASR hypotheses via binary embeddings of entities and words.

Introduction
Reranking models have been shown effective for reducing errors in a variety of Natural Language Processing tasks such as Named Entity Recognition [26,8], Syntactic Parsing [100,101] and Statistical Machine Translation [102]. A reranking model typically treats the baseline system as a black box and is trained to rank the competing hypotheses based on more complex or global information.
In an ASR system, discriminative language model (DLM) is first introduced by Roark et al. [103] for reranking ASR hypotheses. They adopt a single perceptron to modify the confidence scores of hypotheses generated by a baseline ASR system. By using only n-gram features, their reranking model is shown capable of reducing the Word Error Rate (WER) of an ASR system. His work is followed by several variants with a variety of feature choices such as syntactic features [57,58], which try to capture correlation between simple features on the feature level. However, existing DLMs still suffer from poor generalization power and are vulnerable to a shortage of training data, because most of them rely on linear or log-linear models that fail to take into consideration the correlation of input features on the model level. Apart 50 from feature engineering, using hidden variables encoding semantic information helps to improve the generalization power.
Koo et al. [101] proposes a hidden-variable model to rerank syntactic parsing trees.
By linking input features to hidden states corresponding to word senses or classes, they achieve improved accuracy over a linear baseline. Inspired by the success of Koo et al., we propose to use the computational structure of Restricted Boltzmann Machine (RBM) [104] for the task of ASR hypotheses reranking. RBM is a neural network composed of one hidden layer and one input layer. The hidden layer of RBM can be viewed as embeddings of the input and has been shown capable of capturing high-order correlation and semantic information in the context of language modeling [105,106]. These approaches model the probability of a fixed length of word sequences, i.e., N -grams, using only local information, and are trained with a generative objective function. However, RBM cannot be directly used for ASR reranking due to its generative training manner.
In this chapter, we propose two modifications to train RBM in a discriminative manner. We modify the energy function of RBM to incorporate the ASR confidence score, which has been proved critical for reranking by previous DLMs [103,57,58].
We then propose a novel discriminative objective function for training RBM with Nbest lists of ASR hypotheses. Our method differs from existing RBM-based language models [105,106] in two major aspects. Firstly, the proposed RBM reranker is trained discriminatively. Secondly, RBM in our method represents sentences of variable length as global feature vectors. The most attractive property of RBM is that the computational structure is flexible enough to incorporate prior knowledge [107]. As function words have little meanings and are less important for language understanding [108], we decide to focus more on content words. In order to allow the reranking model to have a better knowledge of content words, we propose to fuse the hidden layer with information about a particular class of natural language concepts, i.e., named entities.
For technical simplicity, we constrain the hidden layer of RBM to take only binary values. Therefore, the hidden layer can be seen as a binary embedding of the input.

51
In this regard, the resulting binary embedding encodes the fusion of both word-level and concept-level information.
To our knowledge, this work is the first to consider using hidden layer and fusion of knowledge from both concept and word levels in the context of ASR hypotheses reranking. The remainder of this chapter is structured as follows: Section 2 describes in detail the proposed work; Section 3 shows the empirical results as well as analyses; finally, Section 4 concludes this work and provide more discussion on future work.

Restricted Boltzmann Machine
A Restricted Boltzmann Machine [104] (see Figure 5.1) is a neural network composed of : one n-dimension input feature layer φ(t) = [φ 1 (t), φ 2 (t), · · · , φ n (t)], which is a global feature vector extracted for a raw input t, and one d-dimension binary hidden The joint probability P RBM (t, h) of hidden variables and raw input is defined as where W ∈ R n×d is the matrix specifying the weights of connections between hidden and input layer, and b ∈ R n and c ∈ R d are the bias vectors of the two layers. , and the training objective is to maximize the log likelihood of training data D t∈D ln P RBM (t) 52

Maximum Margin Training for RBM-based Reranker
The goal of generative training of RBM is to learn a probability distribution, which is not necessary for choosing correct ASR hypotheses. Instead, the discriminative training allows the model to explicitly select ASR hypotheses containing fewer errors.
In this section, we describe our discriminatively trained RBM-based reranking model, denoted as dRBM.
Before introducing the training objective function, we first introduce the energy function of RBM model. ASR posterior probabilities produced by the baseline ASR system have been shown useful for reranking in previous works on DLM [103,57,58].
We hence add ASR posterior to the energy function of RBM. The modified energy function is expressed as where P (t|a) is the posterior probability of a given ASR hypothesis t given the acoustic input a, and w 0 is the weight of ASR confidence score fixed during training. We represent each hypothesis as a global feature vector φ(t) using a predefined set of feature functions. In this chapter, we mainly consider using unigram features, yet using more complicated features does not need to change the model. Inspired by the maximum margin training for Bayesian Networks [109], we adopt a discriminative objective function L using likelihood ratio, where D is the training set for the discriminative training of RBM and |D| denotes the number of utterances in training set. GEN(a) refers to the list of N -best hypotheses generated by the baseline ASR system for the acoustic input a, whilet is the oraclebest in the N -best list of t. Intuitively, the learning process finds the parameter setting maximizing the margin between the oracle-best hypotheses and other hypotheses in the N -best list. The subgradient of the objective function is where F(·) is the free energy of RBM defined as Negative: The training algorithm is described in Algorithm 1. For each acoustic input in training set, we select a set of hypotheses T − , which are ranked higher than the oracle best hypothesis. Based on our analysis of the loss function, we boost the score of oracle best with its derivate of negative free energy and penalize hypothesis in T − with their derivates of negative free energy. Note that, as compared with standard RBM training [110], which iterates over input space or samples of input space, our discriminative training needs only to iterate over the N -best list which grows linearly with the size of training data and N -best list.

Training with Prior Knowledge
The binary hidden layer of RBM allows for easily incorporating prior knowledge into the reranking model. We consider using two types of prior knowledge: named entity 54 φ1 φ2 φ3 φ4 h1 h2 h3 φn hd Hidden Layer Input Layer Named Entity Classes … … Figure 5.1: Structure of RBM with Entity-related Prior [9] labels and pre-trained latent layer from texts. Firstly, to improve the capability of recognizing content words, we capture prior of a special class of content words -named entities. As entity related prior also encodes information about word classes, it helps improving the generalization power of language models [111] as well.
Specifically, we extract pairs of named entity words and their classes from texts using a named entity tagger, which annotates the text with 3 widely-adopted named entity classes, i.e., LOCATION, ORGANIZATION and PERSON. As show in Fig. 5.1,   3 variables in the hidden layer of RBM are used to represent named entity classes.
For purpose of reducing ambiguity, we remove words belonging to multiple entity classes. We denote the list of entity-class pairs as G = {w, e}, where w is an index of the unigram feature in the input layer, and e an index of entity-class variable in the hidden layer. The objective function is then augmented with an entity-related regularizer, As introduced in Wang et al. [107], P (h e |φ w ) denotes the probability of a hidden variable h e being activated by a given input feature φ w .
To handle the data sparsity, we initiate connection matrix W of RBM with values pre-trained using a large text corpus and the generative training. The pretraining captures the distributional semantics of input features [112].

Scoring ASR Hypotheses
To score a given hypothesis, we propose two scoring functions using our RBM-based reranker and its combination with SLP. First of all, the RBM-based reranking score S RBM is defined as the logarithm of the unnormalized probabilityP dRBM (t) assigned by the RBM-based reranker solely, As shown above, the re-scoring function is composed of the original ASR posterior, a linear bias, and the hidden variable component. In addition, SLP and RBM are likely to have encoded information complementary to each other due to their different structures and training methods. Therefore, we propose a late fusion of the two methods, which combines their confidence scores in the testing phase. The combined reranking score is where S SLP (t) is the single perceptron based confidence score weighted by α.

Related Work
DLM has been first introduced by Roark et al. in [103], where simple features like N -gram is shown able to effectively reduce WER. This previous work is using a Single Layer Perceptron (SLP) to modify the original posterior probabilities of the outputs of a baseline ASR system using a linear function, where log P (t|a) is the log probability of a word sequence t given the acoustic signal a, and {f i (·)} are the set of feature functions of an utterance weighted by {w i }. Different types of features extracted from syntactic trees [57] and dependency trees [58] have also been used to enrich the feature set.
Apart from feature engineering and using a linear combination of feature functions, inferring hidden variables from the observed input captures semantic information related to word classes and word senses. Our work is closely related to Koo et al. [101] who proposed a hidden-variable model to rerank syntactic parsing trees. For the tractability of their model, they put constrains on the connections between latent variables and visible variables (i.e., input layer) by splitting features into two sets.
However, the way they divide features is specific to syntactic parsing and thus is not applicable to our task. Our model differs from Koo et al. [101] in the sense that the connection is not constrained by their feature type, but instead relying on the structure of RBM to build connections between input and hidden layer.
RBM-based models [105,106] have been explored for language modeling. Both approaches model the probability of a fixed length of word sequences, i.e., N -grams, and trained with a generative objective function. Our method differs from these methods in two major aspects: the training of proposed RBM reranker is discriminative, and it represents sentences of variable length as global feature vectors.

Dataset
We evaluate our work on the latest release of TedLium Corpus [113] which is a set of audio and manually transcribed texts of Ted talks. As shown in Table 5.1. We split the training set of TedLium Version 2 into two parts: former Tedlium Training set Version 1 and the rest. The Version 1 part is a set of 774 Ted talks consisting of 56,800 utterances and more than 1.7 million words, while the remaining of the TedLium training set contains another 718 talks. The evaluation set of our experiment is the testing set of TedLium corpus, which is composed of 11 talks. Our text corpus is the ukWaC corpus [84], which is a collection of texts containing about 1.8 billion words.

Baseline
The baseline ASR system is based on KALDI 1 toolkit [114] including a DNN-based acoustic model. It uses a pre-trained language model 2 , which is released as part of Sphinx project [115] and has achieved a perplexity of 158.3 on a corpus of Ted Talks.
The acoustic model is trained using the training set of TedLium Version 1. The rest of training set of TedLium Version 2 is used for training reranking models.
The baseline SLP reranker is trained by following the work of Lambert et al. [58] that randomly selects K pairs of hypotheses from the N -best lists. Specifically, we randomly select 100 pairs from the 100-best list. We ran the training process is for 10 iterations, as we cannot observe further WER reduction with more iterations.

RBM Setup
We refer to the system integrating prior knowledge with dRBM as p-dRBM. Both dRBM and p-dRBM use 200 hidden units and are trained using the same dataset and 100-best hypotheses as SLP reranker. Since our focus is not on feature engineering, and for simplicity of interpreting our experiment results, we use only unigram features (i.e., single words) in our experiments, which have also been shown as the most effective features by previous works [58]. For training p-dRBM, we first crawled down a set of text summaries of ted talks from Ted website. We then create a list of 20 words for each entity category by tagging the collected text summaries with Stanford named entity recognition tool [116] following description in section 5.2.3. A basic RBM is trained using ukWaC corpus and used for initiating the connection matrix W of p-dRBM. The late fusion of SLP and our proposed methods are denoted as SLP + dRBM and SLP + p-dRBM. We use λ = 0.01 and α = 1.0 as weights of entity-related regularizer and SLP scores in late fusion.

Evaluation
First of all, we analyze the behavior of p-dRBM by computing the most-activated words by p-dRBM as shown in Table 5  These vectors represent how much a word is activated by a given hidden variable. Table 5.3 shows a selected set of hidden variables that can be seen as a set of topics.
We found that RBM can capture meaningful topics by using only sentence-level cooccurrence. We then represent each word as a vector of hidden variables by taking the rows of the matrix W ∈ R |V |×d of p-dRBM.
We rank words based on their cosine similarity with the queries and select the top 5 words for four query words. As shown in Table 5 Since the latent layer of p-dRBM incorporates prior knowledge related to content words (e.g., named entities), it is desirable that the proposed method can better recognize content words, which are more critical for downstream applications such as spoken language understanding. To evaluate the performance of our proposed methods on recognizing content words in a more general way, we words that have higher TF-IDF scores are more likely to be content words. We hence assign more   In particular, p-dRBM is shown working significantly better than other methods on words with medium TF-IDF scores (<5), which is a result of injecting named entity words, e.g., washington (TF-IDF= 3.87), that mostly have TF-IDF between 3.0 and 5.0.

Summary
In this chapter, we proposed an RBM-based reranking model that is discriminatively trained for reranking ASR hypotheses. The proposed reranking learns a binary embedding of words by fusing information of concept-level and word-level in its hidden layer. In comparison with single perceptron based reranker, our proposed approach achieves higher WER reduction. The success of combining single perceptron and RBM-based reranker suggests that two models actually capture complementary information useful for selecting less erroneous ASR hypotheses. In addition, we found that introducing concept-level knowledge to RBM-based reranker results in a better recognition of content words. In the future, we would like to explore the use of lexical knowledge obtained from different resources, e.g., WordNet [117] or SenticNet [15], as additional prior knowledge for the proposed model.

62
Chapter 6 Embedding Concepts for Target-based Sentiment Analysis In this chapter, we focus on leveraging commonsense concepts for target-based sentiment analysis.

Introduction
In the past decade, sentiment analysis [118] has become increasingly popular for processing social media data on online communities, blogs, wikis, microblogging platforms, and other online collaborative media. Sentiment analysis is a branch of affective computing research [119] that aims to classify text into either positive or negative, but sometimes also neutral [120]. Most of the literature is on English language but recently an increasing number of publications is tackling the multilinguality issue [121].
While most works approach it as a simple categorization problem, sentiment analysis is actually a suitcase research problem [2] that requires tackling many NLP tasks, including named entity recognition [8], word polarity disambiguation [122], personality recognition [123], sarcasm detection [124], and aspect extraction. The last one, in particular, is an extremely important subtask that, if ignored, can consistently reduce the accuracy of sentiment classification in the presence of multiple opinion targets.
Hence, aspect-based sentiment analysis (ABSA) [125,126,127] extends the typical setting of sentiment analysis with a more realistic assumption that polarity is associated with specific aspects (or product features) rather than the whole text unit.
For example, in the sentence "The design of the space is good but the service is horrible", the sentiment expressed towards the two aspects ("space" and "service") is completely opposite. Through aggregating sentiment analysis with aspects, ABSA allows the model to produce a fine-grained understanding of people's opinion towards a particular product [128].
Targeted (or target-dependent) sentiment classification [44,40,47], instead, resolves the sentiment polarity of a given target in its context, assuming that a sentence  [48] has attempted to tackle the challenges of both ABSA and targeted sentiment analysis. The task is to jointly detect the aspect category and resolve the polarity of aspects with respect to a given target.
Deep learning methods [42,43,44,45,47] have achieved great accuracy when applied to ABSA and targeted sentiment analysis. Especially, neural sequential models, such as LSTM networks [24], are of growing interest for their capacity of representing sequential information. Moreover, most of these sequence-based methods incorporate the attention mechanism, which has its root in the alignment model of machine translation [129]. Such mechanism takes an external memory and representations of a sequence as input and produces a probability distribution quantifying the concerns in each position of the sequence. (ii) We extend the classic LSTM cell with components accounting for integration with external knowledge; (iii) We fuse affective commonsense concepts with word-level information in a deep neural network.

Methodology
In this section, we describe the proposed attention-based neural architecture in detail: we first proposed the task definition of targeted ABSA, followed by an overview of the whole neural architecture; afterwards, we describe instance attention and global attention model; lastly, we describe the proposed knowledge-embedded extension of LSTM cell.

Task Definition
A sentence s consists of a sequence of words. Similar to [47], we consider all mentions of the same target as a single target. A target t composed of m words in sentence s, denoted as T = {t 1 , t 2 , · · · , t i , · · · , t m } with t i referring to the position of ith word in the target expression, the task of targeted ABSA can be divided into two subtasks.
Firstly, it resolves the aspect categories of t belonging to a predefined set. Secondly, it classifies the sentiment polarity with respect to each aspect category associated with t.

Overview
In this section, we provide an overview of the proposed method. Our neural architecture consists of two components: the sequence encoder and a hierarchical attention component. Afterwards, the target representation together with the aspect embeddings is used for computing a sentence-level attention transforming the whole sentence into a vector.
The sentence-level attention component returns one sentence vector for each aspect and target pair. The aspect-based sentence vector is then fed into the corresponding multi-class (e.g., None, Neural, Negative, and Positive for a 4-class setting; or None, Negative, and Positive for a 3-class setting) classifier to resolve the sentiment polarity.

Long Short-Term Memory Network
The sentence is encoded using an extension of RNN [130], termed LSTM [24], which is firstly introduced by [24 where f i , I i and o i are the forget gate, input gate and output gate, respectively.
where each element of H is a concatenation of the corresponding hidden outputs of both forward and backward LSTM cells.

Target-Level Attention
Based on the attention mechanism, we calculate an attention vector for a target expression. A target might consist of a consecutive or non-consecutive sequence of words, denoted as T = {t 1 , t 2 , · · · , t m }, where t i is the location of an individual word in a target expression. The hidden outputs corresponding to T is denoted as We compute the vector representation of a target t as where the target attention vector α = {α 1 , α 2 , · · · , α m } is distributed over target word sequence T . The attention vector α is a self-attention vector that takes nothing but the hidden output itself as input. The attention vector α of target expression is computed by feeding the hidden output into a bi-layer perceptron, as shown in a ∈ R 1×dm are parameters of the attention component. where the vector β = [β 1 , β 2 , · · · , β L ] is called the sentence-level attention vector.

Sentence-Level Attention Model
Each element β i encodes the salience of the word w i in the sentence s with respect to the aspect a and target T . Existing research on targeted sentiment analysis or ABSA mostly uses targets or aspect terms as queries.
At first, each h i is transformed to a d m dimensional vector by a multi-layer neural network with a tanh activation function, followed by a dense softmax layer to generate a probability distribution over the words in sentence s, i.e., where v a is the aspect embedding of aspect a, H v t is the operation concatenating v t

Commonsense Knowledge
In order to improve the accuracy of sentiment classification, we use commonsense knowledge as our knowledge source to be embedded into the sequence encoder. In particular, we use SenticNet [15], a commonsense knowledge base that contains 100,000 concepts associated with a rich set of affective properties (Table 6.1). These affective properties provide not only concept-level representation but also semantic links to the aspects and their sentiment. For example, the concept 'rotten fish' has property "KindOf-food" that directly relates with aspects such as 'restaurant' or 'food quality', but also emotions, e.g., 'joy', which can support polarity detection (Fig. 6.3).
However, the high dimensionality of SenticNet hinders it from being used in deep neural models. AffectiveSpace [10] has been built to map the concepts of SenticNet to continuous low-dimensional embeddings without losing the semantic and affective relatedness of the original space. We have included a brief introduction of efficiently generating low-dimensional concept embeddings using random projection in Section 2.5.1.2. Based on this new space of concepts, we embed concept-level information into deep neural sequential models to better classify both aspects and sentiment in natural language text.

Sentic LSTM
In order to leverage SenticNet's affective commonsense knowledge efficiently, we propose an affective extension of LSTM, termed Sentic LSTM. It is reasonable to assume that SenticNet concepts contain information complementary to the textual word sequence as, by definition, commonsense knowledge is about concepts that are usually taken for granted and, hence, absent from the text. Sentic LSTM aims to entitle the concepts with two important roles: 1) assisting with the filtering of information flowing from one time step to the next and 2) providing complementary information to the memory cell. At each time step i, we assume that a set of knowledge concept candidates can be triggered and mapped to a d c dimensional space. We denote the set of K concepts as {µ i,1 , µ i,2 , · · · , µ i,K }. First, we combine the candidate embeddings into a single vector as follows: As we realized that there are only up to 4 extracted concepts for each time step, we simply use the average vector (although a more sophisticated attention model can also be easily employed to replace the averaging function).
Our affective extension of LSTM is illustrated in Equation 6.7. At first, we assume that affective concepts are meaningful cues to control the information of token-level information. For example, a multi-word concept 'rotten fish' might indicate that the word 'rotten' is a sentiment-related modifier of its next word 'fish' and, hence, less information should be filtered out at next time step. We thus add knowledge concepts to the forget, input, and output gate of standard LSTM to help to filter the information. The presence of affective concepts in the input gate is expected to prevent the memory cell from being affected by input tokens conflicting with pre-existing knowledge. Similarly, the output gate uses such knowledge to filter out irrelevant information stored in the memory.
Another important feature of Sentic LSTM is based on the assumption that the information from the concept-level output is complementary to the token level.
Therefore, we extended the regular LSTM with an additional knowledge output gate o c i to output concept-level knowledge complementary to the token-level memory.
Since AffectiveSpace is learned independently, we leverage a transformation matrix W c ∈ R d h ×dµ to map it to the same space as the memory outputs. In other words, o c i models the relative contributions of token level and concept level.
Moreover, we notice that o c i * tanh(W c µ i ) actually resembles the functionality of the sentinel vector used by [53], which allows the model to choose whether to use affective knowledge or not.

Prediction and Parameter Learning
The objective to train our classier is defined as minimizing the sum of the cross-entropy losses of prediction on each target-aspect pair, i.e., where A is the set of predefined aspects, and p a c,t is the probability of the gold-standard polarity class c given target t with respect to a sentiment category a, which is defined by a softmax function, where W p and b a s are the parameters to map the vector representation of target t to the polarity label of aspect a. To avoid overfitting, we add a dropout layer with dropout probability of 0.5 after the embedding layer. We stop the training process of our model after 10 epochs and select the model that achieves the best performance on the development set.

Dataset and Resources
We evaluate our method on two datasets: SentiHood [48] and a subset of Semeval 2015 [131]. SentiHood is built by querying Yahoo! Answers with location names of London city. To inject the commonsense knowledge, we use a syntax-based concept parsergithub.
com/senticnet to extract a set of concept candidates at each time step, and use Af-fectiveSpacesentic.net/downloads as the concept embeddings. In case no concepts are extracted, a zero vector is used as the concept input.

Experiment Setting
We evaluate our method on two sub-tasks of targeted ABSA: 1) aspect categorization and 2) aspect-based sentiment classification. Following Saeidi et al. [48], we treat the outputs of aspect-based classification as hierarchical classes. For aspect categorization, we output the label (e.g., in the 3-class setting, it outputs 'Positive', 'Negative', or 'None') with the highest probability for each aspect. For aspect-based sentiment classification, we ignore the scores of 'None'. For evaluating the aspect-based sentiment classification, we simply calculate the accuracy averaged over aspects. We evaluate aspect categorization as a multi-label classification problem so that results are averaged over targets instead of aspects.
We evaluate our methods and baseline systems using both loose and strict metrics.
We report scores of three widely used evaluation metrics of multi-label classifier: Macro-F1, Micro-F1, and strict Accuracy. Given the dataset D, the ground-truth aspect categories of the target t ∈ D is denoted as Y t , while the predicted aspect categories denoted as Y t . The three metrics can be computed as is an indicator function.  • Micro-F1 = 2 Mi-P×Mi-R Mi-P+Mi-R , which is based on Micro-Precision (Mi-P) and Micro-Recall (Mi-R), where Mi-P= t∈D |Yt∩ Yt| t∈D Yt , and Mi-R= t∈D |Yt∩ Yt| t∈D Yt .

Performance Comparison
We compare our proposed method with the methods that have been proposed for targeted ABSA as well as methods proposed for ABSA or targeted sentiment analysis but applicable to targeted ABSA.
Furthermore, we also compare the performances of several variants of our proposed method in order to highlight our technical contribution. We run the model for multiple times and report the results that perform best in the development set. For  Semeval-2015 dataset, we report the results of the final epoch.
• TDLSTM: TDLSTM [44] adopts Bi-LSTM to encode the sequential structure of a sentence and represents a given target using a vector averaged on the hidden outputs of target instances.
• LSTM + TA: Our method learns an instance attention on top of the outputs of LSTM to model the contribution of each instance.
• LSTM + TA + SA: In addition to target instance attention, we add a sentence-level attention to the model.
• LSTM + TA + DMN SA: The sentence-level attention is replaced by a dynamic memory network with multiple hops [45]. We run the memory network with different numbers of hops and report the result with 4 hops (best performance on development set of SentiHood). We exclude the case of zero hops as it corresponds to Bi-LSTM + TA + SA.
• LSTM + TA + SA + KB Feat: Concepts are fed into the input layer as additional features.
• LSTM + TA + SA + KBA: This is an integration of the method proposed by [53], which learns an attention over the concept embeddings (the embeddings are combined with the hidden output before being fed into the classifier).
• Recall LSTM + TA + SA: LSTM is extended with a recall knowledge gate as in [54].
• Sentic LSTM + TA + SA: The encoder is replaced with the proposed knowledge-embedded LSTM.
The word embedding of the input layer is initialized by a pre-trained skip-gram model [27] with 150 hidden units on a combination of Yelp 1 and Amazon review dataset [132] and 50 hidden units for the bi-directional LSTM. However, it is notable that, on the Semeval-2015 dataset, the improvement is relatively smaller. We conjecture the reason is that SentiHood has masked the target as a special word "LOCATION", which resulted less informative than the full name of aspect targets that are used by Semeval-2015.

Results of Attention Model
Hence, using only the hidden outputs regarding the target does not suffice to represent the sentiment of the whole sentence in SentiHood. Compared with target averaging model, the target-level attention achieves some improvement (even though not significant), as the target attention is capable of identifying the part of target expressions with higher sentiment salience. On the other hand, it is notable that the two-step attention achieves significant improvement on both aspect categorization and sentiment classification, indicating that the target-and aspect-dependent sentence attention could retrieve information relevant to both tasks.
To our surprise, using multiple hops in the sentence-level attention fails to bring in any improvement. The performance even falls down significantly on Semeval-2015 with a much smaller number of training instances but larger aspect set than Senti-Hood. We conjecture the reason is that using multi-hops increases the number of parameter to learn, which makes it less applicable to small and sparse datasets.

Visualization of Attention
We visualize the attention vectors of sentence-level attention in Figure 6.4 with respect to "Transition-location" and "Price" aspects. The two attention vectors have encoded quite different concerns in the word sequence.
In the first example, the 'Transition-location' attention attends to the word "long", which is expressing a negative sentiment towards the target. In comparison, the 'Price' attention attends more to the word 'cheap', which is related to the aspect. That is to say, the two attention vectors are capable of distinguishing information related to different aspects. As visualized in Figure 6.5, the target-level attention is capable of selecting the part of target expression of which the aspect or sentiment is easier to be resolved.

Results of Knowledge-Embedded LSTM
It can be seen from Table 6.3 and 6.4 that injecting knowledge into the model improves the performance in general. Since AffectiveSpace encodes the information about affective properties that are semantically related to the aspects, it is reasonable to find out that it can improve performance on both tasks. The results show that our proposed Sentic LSTM outperforms baseline methods, even if not significantly.
An important outcome of the experiments is that Sentic LSTM significantly outperforms a baseline (LSTM + TA + SA + KB feat) feeding the knowledge features to the input layer, which confirms the efficacy of using a knowledge output gate to control the flow of background knowledge. Furthermore, the superior performance of Sentic LSTM over Recall LSTM and KBA indicates that the activated knowledge concepts can also help filtering the information that conflicts with the background knowledge.

AffectiveSpace v.s. SentiWordNet
At last, we compare systems using different sentiment knowledge bases. SentiWordNet is a lexicon-based sentiment knowledge base consisting of word sense synsets annotated with sentiment polarity. However, it is notable that SentiWordNet contains  Negative) to emphasize on the sentiment-bearing words. Table 6.5 shows the comparison of the two knowledge base. We report the results using our best-performed system (Sentic LSTM + TA + SA). It shows that using the AffectiveSpace achieves superior performance than using SentiWordNet. We conjecture the reason is that the word sense synsets is not as informative as affective properties. Moreover, probably because the link between word senses and aspects are not straightforward, we find the gap in sentiment classification is smaller than aspect categorization.

Summary
In this chapter, we propose a neural architecture for the task of targeted ABSA. We explicitly modeled the attention as a two-step model which encodes targets and full sentence. The target-level attention learns to attend to the sentiment-salient part of a target expression and generates a more accurate representation of the target, while the sentence-level attention searches for the target-and aspect-dependent evidence over the full sentence. Moreover, we propose an extension of the LSTM cell so that it could more effectively incorporate affective commonsense knowledge when encoding the sequence into a vector. In the future, we would like to collectively analyze the sentiment of multiple targets co-occurring in the same sentence and investigate the role of commonsense knowledge in modeling the relation between targets.

Summary and Future Work
In this final chapter, we summarize our contribution to the problem of leveraging concept-level information for natural language processing, followed by possible directions of research that we would like to explore in the future.

Summary of Proposed Methods
Concepts are playing a critical role in natural languages, especially in the context of opinion understandings. In this thesis, we address the problem of leveraging conceptlevel information for concept-centered natural language processing tasks that are core modules in a pipeline opinion understanding system with a particular focus on fusing information from multiple levels. We identify our main contribution as follows: • In Chapter 3, we demonstrate that incorporating entity-based features into the learning process of word embedding could better encode entity-related information in the word embedding space. We present a novel method to learn word embeddings in a multi-task setting, where entity-level information is fused with the word-level by setting the training objective to predicting the entity-related features. Each entity-related feature type is modeled as a separate subtask of prediction, while the word embedding layer is shared by all subtasks. We then demonstrate the effectiveness of the resulting word embedding by feeding word embeddings as features into a CRF-based NER tagger.
• In Chapter 4, we capture the semantic correlation between fine-grained entity types. We propose a process to represent the entity labels with their prototpyes.
The process can be implemented via either manual selection or an automated algorithm based on PMI. At the core of our proposed method are word embeddings learned from a large corpus. The embedding of each label is then defined as the average word embedding of its prototypes. In doing so, we able to project the entity labels into the word embedding space that has encoded distributional semantics. We validate the proposed label embedding with two different settings of fine-grained named entity typing: 1) few-shots learning, where all labels are assumed having presented in the training data; and 2) zero-shot learning, where the second level entity types are assumed absent from the training set.
Our experiment shows that the resulting label embedding outperforms existing methods in both few-shots and zero-shots learning setting.
• In Chapter 5, we study the problem of using entity-type information to help to rank ASR hypotheses. We propose an RBM based ASR reranking model that can be trained using discriminative objective function. The hidden layer of RBM are in fact binary embeddings of the input. We design a regularization term to softly constrain the binary embeddings such that it can encode a better knowledge of entity types. The resulting binary embedding could also be deemed as a fusion of entity-related prior information and word-level input.
Our experiment shows that the proposed ASR reranking model can outperform the popularly used single-perceptron baseline thanks to a better knowledge of natural language concepts.
• In Chapter 6, we address the problem of targeted sentiment analysis which is an important subtask of opinion understanding. Our solution leveraged commonsense concepts via an extended LSTM network. Our contribution also includes a two-layer attention neural architecture that jointly learns the representation of the targeted entity as well as a target-aware representation of the whole sentence. More specifically, our framework explicitly fuses the concept-level and word-level information via a separate output gate of LSTM cell. The experiment shows that the knowledge of AffectiveSpace provides complementary semantics 82 to assist resolving both aspect categories as well as the corresponding sentiment polarity.
We further summarize the results of our work into Table 7.1. Improve FNET at both few-shots and zero-shot setting.
RBM-based reranking model Incorporate the conceptlevel prior knowledge into the binary embedding layer of RBM.
Improve ASR reranking algorithm using concept-level prior.
Sentic-LSTM Utilize commonsense knowledge for targeted sentiment analysis; propose an effective extension of LSTM to incorporate the concept-level and word-level inputs.
Improve both aspect categorization and polarity detection of targeted sentiment analysis At last, the results presented in this thesis suggests that the concept-level knowledge is generally helpful as long as concepts (including named entities) are playing a key roles in the task. On the other hand, as concepts are typically of high dimensionality, the embedding layer together with efficient computational structure could make the concept-level information more accessible.

Future Work
We would discuss possible directions we like to explore in the future.

Limitation of Our Work
One limit of our solution to leveraging concepts as well as fusing concept-level and word-level information is that our usage of concepts is constrained to the original form of input data. In other words, the use of concepts does not change the form of inputs that are mostly fed into the model as a sequence of words. However, when considering the interaction between concepts, keeping to the original sequential order of words might become redundant or sometimes might even cause unnecessary confusion. For example, for a sentence 'this place is safe, but areas close to it are very dangerous', if the target is to analyze the sentiment towards the first place, the second clause might be cause confusion if being processed in its original order. In contrast, parsing the sentence into concepts and relations is able to shorten the sentence into a simple tuple (place,be,safe) which could not only help remove the redundant information but also create the dependency between a target and a correct sentiment-related modifier. Inspired by previous work reforming the sequential input into structures such as trees [133] or graphs [134], we would like to investigate exploiting non-sequential structure to better perform concept-level analysis of natural language.

Graph-based Concept Embedding
Concepts are naturally organized in graphs. For example, ConceptNet defines a set of relations such as Isa relation between concepts. A graph can be built with concept being the nodes and relations being the edges. Similarly, AffectiveSpace can also be converted into a directed graph, even though it uses affective properties (relation + argument). So far, the concept embeddings [10] are built by merely considering decomposition of the concept-property matrix. In other words, the result embedding of concepts are only affected by immediate neighbors.
Recent work on embedding graphs [135] has shed light upon propagating information over a graph so that the embedding of a node is able to also reflect the properties of its indirect neighbors. We could try to infer concept embeddings via the graph structure. Different from existing work, our method will also consider the relation types which are also important for modeling information propagation in AffectiveSpace. 84

Concept-oriented Graph Model
Since concepts are very informative and their interaction is key to properly understanding the meaning of natural language, our first proposal for future work is to extend the usage of concepts to reforming the input sequence. One way to do so is to transform the input to local graphs of concepts by considering their explicit or implicit relations. Explicit relations can be the semantic relation given by knowledge bases such as ConceptNet or SenticNet. Implicit relations, for example, can be the presence of concepts in each other's contexts. By converting the word sequence into concept graphs, the NLP model can explicitly avoid affected by redundant and misleading structures and take into account the interaction between concepts.
In addition, another extension is to consider the global interaction of concepts.
This can be fairly critical tor tasks such as sentiment analysis of geographical entities [48] 1 . For example, people might share the same safety concerns with neighboring locations. A graph built to represent the relation of concepts allows us to perform collective inference via approaches such as label propagation [136].

New Sentiment Analysis Application using Concepts
In future, we also would like to explore new sentiment-related applications using concepts. It is known that sentiment is playing an important role in expressing opinions.
In addition to decoding (or interpreting) people's opinion, it is also of our interest to generate sentiment-aware responses. Especially, in the context of e-commerce, automatically generating sentiment-aware responses to customers' reviews would largely improve the customer experience. Moreover, rather than generating a sequence of words, we are more interested in generating a mix of words and commonsense concepts. We believe that, by leveraging commonsense concepts as well as concept relations, it could help improve the consistency and comprehensiveness of generated responses.
1 Similar to named entities, we could also consider geographical entities as a type of concepts.