{'fraction_non_alphanumeric': 0.034038146774893946, 'fraction_numerical': 0.009753816347663426, 'mean_word_length': 4.569860465116279, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Sarcasm Target Identification: Dataset and An Introductory Approach

Past work in computational sarcasm deals primarily with sarcasm detection. In this paper, we introduce a novel, related problem: sarcasm target identification (i.e., extracting the target of ridicule in a sarcastic sentence). As a benchmark, we introduce a new dataset for the task. This dataset is manually annotated for the sarcasm target in book snippets and tweets based on our formulation of the task. We then introduce an automatic approach for sarcasm target identification. It is based on a combination of two types of extractors: one based on rules, and another consisting of a statistical classifier. Our introductory approach establishes the viability of sarcasm target identification, and will serve as a baseline for future work.


Introduction
Sarcasm is a form of verbal irony that is intended to express contempt or ridicule (Source: The Free Dictionary). While several approaches have been reported for sarcasm detection (Rajadesingan et al., 2015;Joshi et al., 2015;González-Ibánez et al., 2011), no past work, to the best of our knowledge, has attempted to identify a crucial component of sarcasm: the target of ridicule (Campbell and Katz, 2012). This is important because the sentiment of the sarcastic text needs to be attributed to this target of ridicule. Towards this motivation, we introduce 'sarcasm target identification': the task of extracting the target of ridicule (i.e., sarcasm target) of a sarcastic text. The input is a sarcastic text while the output is either (a) a subset of words in the sentence that point to the sarcasm target, or (b) a fall-back label 'Outside' 1 . In this paper, we present a manually labeled dataset consisting of text from two domains: tweets and book snippets. We also report an automatic approach that takes as input a sarcastic text and returns its sarcasm target. Our hybrid approach combines a rule-based extractor (that implements a set of rules) and a statistical extractor (that uses a word-level classifier for every word in the sentence, to predict if the word will constitute the sarcasm target). Since this is the first work in sarcasm target detection, no past work exists to be used as a baseline. Hence, we devise two baselines to validate the strength of our work. The first is a simple, intuitive baseline to show if our approach (which is computationally more intensive than this simple baseline) holds value. In absence of past work, using simple and obvious techniques to solve a problem have been considered as baselines in sentiment analysis (Tan et al., 2011;Pang and Lee, 2005). As the second baseline, we use a technique reported for sentiment/opinion target identification since sentiment target identification appears to be related to sarcasm target identification, on the surface. Our manually labeled datasets are available for download at: https://github.com/Pranav-Goel/ Sarcasm-Target-Detection. Each unit consists of a piece of text (either book snippet or tweet) with the annotation as the sarcasm target where the sarcasm target is a subset of words in the text or the fall-back label 'Outside'. In addition to this, our hybrid approach for sarcasm target identification will serve as a baseline for future work. Sarcasm target identification can benefit natural language generation and sentiment analysis systems. Being able to recognize the entity towards which the negative sentiment was intended, a natural language generation system will have more context to generate a response. Similarly, a sentiment analysis system will be able to attribute the negative sentiment in a sarcastic text towards the correct aspect of a product or the appropriate entity.

Related work
Computational sarcasm primarily focuses on sarcasm detection: classification of a text as sarcastic or non-sarcastic. Joshi et al. (2016a) present a survey of sarcasm detection approaches. They observe three trends in sarcasm detection: semi-supervised extraction of sarcastic patterns, use of hashtag-based supervision, and use of contextual information for sarcasm detection Davidov et al., 2010;Joshi et al., 2015). However, to the best of our knowledge, no past work aims to identify phrases in a sarcastic sentence that indicate the target of ridicule in the sentence. Related to sarcasm target identification is sentiment target identification. Sentiment target identification deals with identifying the entity towards which sentiment is expressed in a sentence. Qiu et al. (2011) present an approach to extract opinion words and targets collectively from a dataset. Aspect identification for sentiment has also been studied. This deals with extracting aspects of an entity (for example, color, weight, battery in case of a cell phone). Probabilistic topic models have been commonly used for the same. Titov et al. (2008) present a probabilistic topic model that jointly estimates sentiment and aspect in order to achieve sentiment summarization. Lu et al. (2011) perform multi-aspect sentiment analysis using a topic model.

Example Target
Love when you don't have two minutes to send me a quick text. you Don't you just love it when Microsoft tells you that you're spelling your own name wrong.
Microsoft I love being ignored. being ignored He is as good at coding as Tiger Woods is at avoiding controversy.
He, Tiger Woods Oh, and I suppose the apple ate the cheese. Outside Table 1: Examples of sarcasm targets Several other topic model-based approaches to aspect extraction have been reported (Mukherjee and Liu, 2012). To the best of our knowledge, ours is the first work that deals with sarcasm target identification.

Formulation
Sarcasm is a well-known challenge to sentiment analysis (Pang et al., 2008). Consider the sarcastic sentence 'My cell phone has an awesome battery that lasts 20 minutes'. This sentence mocks the battery of the cell phone. Aspectbased sentiment analysis deals with identifying sentiment expressed towards different aspects or dimensions of an entity. Therefore, aspect-based sentiment analysis needs to identify that the sentence expresses a negative sentiment towards the aspect 'battery'. With sarcasm target identification, we hope to enable aspect-based sentiment analysis to attribute the negative sentiment to the correct target aspect in the case of a sarcastic text. We define the sarcasm target as the entity or situation being ridiculed in a sarcastic text. In the case of 'Can't wait to go to class today', the word 'class' is the sarcasm target. We make two (fair) assumptions here. (a) Every sarcastic text has at least one sarcasm target. This holds true by definition of sarcasm. Also, (b) The notion of sarcasm target is applicable for sarcastic texts only. A non-sarcastic text does not have a sarcasm target. With these assumptions, we define sarcasm target identification as follows. Given a sarcastic text, sarcasm target identification is the task of extracting the subset of words that indicate the target of ridicule. However, in some cases, the target of ridicule may not be present among the words. In such a case, a fall-back label 'Outside' is expected. Examples of some sarcasm targets are given in Table 1. Some challenges of sarcasm target identification are: • Presence of multiple candidate phrases: Consider the sentence 'This phone heats up so much that I strongly recommend chefs around the world to use it as a cook-top'. In this sentence, the words 'chefs', 'cooktop' and 'phone' are candidate phrases. However, only the 'phone' is being ridiculed in this sentence.
• Multiple sarcasm targets: A sentence like 'You are as good at coding as he is at cooking' ridicules both 'you' and 'he', and hence, both are sarcasm targets.
• Absence of a sarcasm target word (the 'Outside' case): Consider something bad happens in the beginning of the day and one says, 'What a great way to start off the day!'. No specific word in the sentence is the sarcasm target. The target here is the situation that started off the day. We refer to such cases as the 'Outside' cases.

Collection of sarcastic text
We experiment with two datasets: book snippets and tweets. The dataset of book snippets is a sarcasm-labeled dataset by (Joshi et al., 2016b). From this dataset, 224 book snippets marked as sarcastic are used. The second dataset is dataset of tweets, given by (Riloff et al., 2013). 506 sarcastic tweets from this dataset are used. These book snippets and tweets are manually annotated with the sarcasm target. The statistics of the two datasets are shown in Table 2. The average length of a sarcasm target is 1.6 words in the case of book snippets and 2.08 words in the case of tweets. The last two rows in the table point to an interesting observation. In both the datasets, the average polarity strength 2 of sarcasm target is lower than polarity strength of rest of the sentence. This shows that the sarcasm target is likely to be more neutral than sentiment-bearing.

Annotation
The annotation is manually carried out by three annotators. The annotators hold more than 5 years of linguistic annotation experience each for sentiment analysis, word sense disambiguation and related tasks. The annotators first discuss the notion of sarcasm target. They are then given the following guiding question for their annotation: 'The given text is sarcastic. Which words in this text indicate the target of sarcasm that the author making fun of? If you cannot locate specific words, mark them as 'Outside''. The target could be an entity or a phrase referring to the situation, and the annotators were told to prefer specific entities over situations when possible. Every textual unit in both the datasets is labeled for sarcasm target, and the label comprises of either a subset of words in the tweet or the fall-back label 'Outside'. Note that a subset of words means that the word or phrase is taken exactly as it is from the sarcastic text being marked for the target -no change in the case, punctuation or wording occurs when the corresponding labels are given. In the case of a phrase, any punctuation at the boundary of the phrase is not included in the sarcasm target/label. In the case of multiple targets, all the targets are annotated.

Inter-annotator agreement
To understand the challenge posed by sarcasm target annotation, we conduct an additional experiment. Two annotators, different from the original set of expert annotators who produced the gold labels, are given the same task of annotation. The annotators are undergraduate students of computer science with English as the primary language of instruction throughout their academic tenure. Following the question as stated above, the new annotators annotate a subset of 50 tweets and 50 book snippets separately from the original annotation and separately from each other. Since this is a phrase extraction task and not a label assignment task, we use match two proportion-based metrics to quantify the inter-annotator agreement. The first metric is exact match. This metric is defined as the percentage of texts for which the pair of annotations are exactly the same. in the case of the book snippets dataset, the exact match is 28% while in the case of tweets, it is 38%. The higher value in the case of tweets is likely to be due to the more direct nature of ridicule in tweets leading to clearer and more specific targets. The second metric is partial match. This metric is defined as the percentage of texts for which the pair of annotations show a partial overlap. The partial match in the case of both the book snippets and tweets datasets is 66%. There was no overlap in annotations for about one third of the dataset for both book snippets and tweets. The values of exact and partial match hint at the task presenting a reasonable challenge to humans.

Architecture
Our introductory approach for sarcasm target identification is depicted in Figure 1. The input is a sarcastic sentence while the output is the sarcasm target. The approach consists of two kinds of extractors: (a) a rule-based extractor that implements nine rules to identify different kinds of sarcasm targets, and (b) a statistical extractor that uses statistical classification techniques. The two extractors individually generate lists of candidate sarcasm targets. The third component is the integrator that makes an overall prediction of the sarcasm target by choosing among the sarcasm targets returned by the individual extractors. The overall output is a subset of words in the sentence. In case no word is found to be a sarcasm target, a fall-back label 'Outside' is returned. In the forthcoming subsections, we describe the three modules in detail.

. Rule-based Extractor
Our rule-based extractor consists of nine rules that take as input the sarcastic sentence, and return a set of candidate sarcasm targets. The rules are summarized in Table 3. In detail, these rules are as follows: 1. R1 (Pronouns and Pronominal Adjectives): R1 returns pronouns such as 'you, she, they' and pronominal adjectives (followed by their object) (as in the case of 'your shoes'). Thus, for the sentence 'I am so in love with my job', the phrases 'I' (pronoun) and 'my job' (based on the pronominal adjective 'my') are returned as candidate sarcasm targets. This is based on observations by Shamay et al. (2005).
2. R2 (Named Entities): Named entities in a sentence may be sarcasm targets. This rule returns all named entities in the sentence. In case of 'Olly Riley is so original with his tweets', R2 predicts the phrase 'Olly Riley' as a candidate sarcasm target.
3. R3 (Sentiment-bearing verb as the pivot): This rule is based on the idea by (Riloff et al., 2013) that sarcasm may be expressed as a contrast between a positive sentiment verb and a negative situation. In case of 'I love being ignored', the sentiment-bearing verb 'love' is positive. The object of 'love' is 'being ignored'. Therefore, R3 returns 'being ignored' as the candidate sarcasm target. If the sentiment-bearing verb is negative, the rule returns 'Outside' as a candidate sarcasm target.
4. R4 (Non-sentiment-bearing verb as the pivot): This rule applies in case of sentences where the verb does not bear sentiment. The rule identifies which out of subject or object has a lower sentiment score, and returns the corresponding portion as the candidate sarcasm target. For example, rule R4 returns 'to have a test on my birthday' as the candidate sarcasm target in case of 'Excited that the teacher has decided to have a test on my birthday!' where 'decided' is the nonsentiment-bearing verb. This is also based on Riloff et al. (2013).

Rule Definition
Example

R1
Return pronouns and pronominal adjectives Love when you don't have two minutes to send me a quick text .. ; I am so in love with my job.

R2
Return named entities as target Don't you just love it when Microsoft tells you that you're spelling your own name wrong.

R3
Return direct object of a positive sentiment verb I love being ignored.

R4
Return phrase on lower sentiment side of primary verb So happy to just find out it has been decided to reschedule all my lectures and tutorials for me to night classes at the exact same times! R5 Return Gerund and Infinitive verb phrases Being covered in hives is so much fun! R6 Return nouns preceded by a positive sentiment adjective Yep, this is indeed an amazing donut ..

R7
Return subject of interrogative sentences A murderer is stalking me. Could life be more fun? R8 Return subjects of comparisons (similes) He is as good at coding as Tiger Woods is at controversies.

R9
Return demonstrative adjective-noun pairs Oh, I love this jacket! Combining the outputs of individual rules to generate candidate sarcasm targets of the rule-based extractor: To generate the set of candidate sarcasm targets returned by the rule-based extractor, a weighted majority approach is used as follows. Every rule above is applied to the input sarcastic sentence. Then, every word is assigned a score that sums the accuracy of rules which predicted that this word is a part of the sarcasm target. This accuracy is the overall accuracy of the rule as determined by solely the rule-based classifier. Thus, the integrator weights each word on the basis of how good a rule predicting it as a target was. Words corresponding to the maximum value of this score are returned as candidate sarcasm targets.

Statistical Extractor
The statistical extractor uses a classifier that takes as input a word (along with its features) and returns if the word is a sarcasm target. To do this, we decompose the task into n classification tasks, where n is the total number of words in the sentence. This means that every word in input text is considered as an instance, such that the label can be 1 or 0 depending on whether or not the given word is a part of sarcasm target. For example, 'Tooth-ache is fun' with sarcasm target as 'tooth-ache' is broken down into three instances: 'tooth-ache' with label 1, 'is' with label 0 and 'fun' with label 0. In case the target lies outside the sentence, all words have the label 0. We then represent the instance (i.e., the word) as a set of following features: (A) Lexical: Unigrams, (B) Part of Speech (POS)-based features: Current POS, Previous POS, Next POS, (C) Polarity-based features: Word Polarity : Sentiment score of the word, Phrase Polarity : Sentiment score for the trigram formed by considering the previous word, current word and the next word together (in that order). These polarities lie in the range [-1,+1]. These features are based on our analysis that the target phrase or word tends to be more neutral than the rest of the sentence, and (D) Pragmatic features: Capitalization : Number of capital letters in the word. Capitalization features are chosen based on features from . The classifiers are trained with words as instances while the sarcasm target is to be computed at the sentence level. Hence, the candidate sarcasm target returned by the statistical extractor consists of words for which the classifier returned 1. For example, the sentence 'He is nice' is broken up into three instances: 'He', 'is' and 'nice'. If the classifier returns 1, 0, 0 for the three instances respectively, the statistical extractor returns 'He' as the candidate sarcasm target. Similarly, if the classifier returns 0, 0, 0 for the three instances, the extractor returns the fall-back label 'Outside'.

Integrator
The integrator determines the sarcasm target based on the outputs of the two extractors. We consider two configurations of the integrator: 1. Hybrid OR: In this configuration, the integrator predicts the set of words that occur in the output of either of the two extractors as the sarcasm target. If the lists are empty, the output is returned as 'Outside'.
2. Hybrid AND : In this configuration, the integrator predicts the set of words that occur in the output of both the two extractors as the sarcasm target. If the intersection of the lists is empty, the output is returned as 'Outside'.
The idea of using two configurations OR and AND is based on a rule-based sarcasm detector by (Khattri et al., 2015). While AND is intuitive, the second configuration OR is necessary because our extractors individually may not capture all forms of sarcasm target. This is intuitive because our rules may not cover all forms of sarcasm targets.

Experiment Setup
We use SVM Perf (Joachims, 2006) to train the classifiers, optimized for F-score with epsilon e=0.5 and RBF kernel 3 . We set C=1000 for tweets and C=1500 for snippets. We report our results on four-fold cross validation for both datasets. Note that we convert individual sentences into words. Therefore, the dataset in the case of book snippets has 6377 instances, while the one of tweets has 6610 instances. The four folds for cross-validation are created over these instances. With a word as instance, the task is binary classification: 1 indicating that the word is a sarcasm target and 0 indicating that it is not. For rules in the rule-based extractor, we use tools in NLTK (Bird, 2006), wherever necessary. We consider two baselines with which our hybrid approach is compared: 1. Baseline 1: All Objective Words: As the first baseline, we design a naïve approach for our task: include all words of the sentence which are not stop words, and have neutral sentiment polarity, as the predicted sarcasm target.
2. Baseline 2: Sequence labeling has been reported for opinion target identification (Jin et al., 2009). Therefore, we use SVM-HMM (Altun et al., 2003) with default parameters as the second baseline.
We report performance using two metrics: Exact Match Accuracy and Dice Score. These metrics have been used in past work in information extraction (Michelson and Knoblock, 2007). As per their conventional use, these metrics are computed at the sentence level. The metrics are described as: • Exact Match (EM) Accuracy: An exact match occurs if the list of predicted target(s) is exactly the same as the list of actual target(s). The accuracy is computed as number of instances with exact match divided by total instances.
• Dice Score (DC): Dice score (Sørensen, 1948) is used to compare similarity between two samples. This is considered to be a better metric than Exact match 3 RBF Kernel performed better than linear kernel. accuracy because it accounts for missing words and extra words in the target. Let the two lists (predicted and actual) be X and Y. Dice score is given by (2X ∩ Y )/(X + Y ).   This section presents our results in two steps: performance of individual rules that are a part of the rule-based extractor, and performance of the overall approach.

Rule
5.3.1. Performance of rules in the rule-based extractor Tables 4 and 5 present the performance of the rules in our rule-based extractor, for snippets and tweets respectively. The two metrics (exact match accuracy and dice score) are reported for two cases: Overall and Conditional. 'Overall' spans all text units in the dataset whereas 'Conditional' is limited to text units which match a given rule (i.e., where the given linguistic phenomenon of, say, gerunds, etc. is observed). Considering the 'Conditional' case is crucial because a rule may be applicable for a specific form of sarcasm target, but may work accurately in those cases. Such a rule will have a low 'overall exact match/dice score' but a high 'conditional exact match/dice score.' Values in bold indicate the best performing rule for a given performance metric. As seen in the tables, the values for 'conditional' are higher than those for 'Overall'. For example, consider rule R7 in Table 4. Exact match of 3.12 (for overall accuracy) as against 9.46 (for conditional accuracy). This situation is typical of rule-based systems where rules may not cover all cases but be accurate for situations that they do cover. For tweets, R3 has a very high dice score (conditional) (55.77). This rule is based on the intuition in (?) that contrast of positive sentiment with negative situation is a strong indicator of sarcasm target. Tables 6 and 7 compare the six approaches for snippets and tweets respectively. All our approaches outperform the baseline in the case of exact match and dice score. in the case of tweets, Table 7 shows that the rule-based extractor achieves a dice score of 29.13 while that for statistical extractor is 31.8. Combining the two together (owing to our hybrid architecture) improves the dice score to 39.63. This improvement also holds for book snippets. This justifies the 'hybrid' nature of our approach. Hybrid OR performs the best in terms of Dice Score. However, for exact match accuracy, Hybrid AND achieves the best performance (16.51 for snippets and 13.45 for tweets). This is likely because Hybrid AND is restrictive with respect to the predictions it makes for individual words. The statistical extractor performs better than rule-based extractor for all three metrics. For example, in the case of tweets, the dice score for statistical extractor is 31.8 while that for rulebased extractor is 29.13. Also, nearly all results (across approaches and metrics) are higher in the case of tweets as compared to snippets. Since tweets are shorter than snippets (as shown in Table 2), it is likely that tweets are more direct in their ridicule as compared to snippets. We discuss an experiment to validate this observation in Section 5.4.. Thus, tackling the task of sarcasm target identification and the new dataset we present can help gain insights into the nature of sarcasm.

Study on nature of sarcasm in tweets versus book snippets
As an example of the kind of investigations into the nature of sarcasm that this dataset can facilitate, we conduct an experiment to test the hypothesis that 'tweets are more direct in their ridicule than snippets'. Book snippets and tweets containing the word 'man' are selected. Thirteen tweets contain the word 'man'. Many book snippets contain the word 'man', of which 13 are randomly chosen among these. 13 tweets and 13 snippets are randomly paired up. Two human annotators (not the ones involved in the sarcasm target annotations) are asked to choose which of the two was more 'direct' in its ridicule. The two human annotators are 25-30 years old, one male engineer (A1) and one female linguist (A2). The two annotators have no prior experience in sarcasm annotation but have studied English as a primary language from school onwards. The two annotators are not told about the claim to be verified, and are not told that one of the pair is a tweet and one of them a book snippet. For every pair, the annotator answers the question "Which of the two is more direct in its ridicule?". A2 selects tweets to be more direct in 11 cases out of 13 while A1 does so in 10 out of 13. In other words, in 11 out of 13 cases, A2 states that for a given pair, the tweet is more direct in its ridicule than the book snippet.

Error analysis
We now discuss sources of errors made by our system.  -For the sentence 'I love being ignored', we can only infer that the act of "being ignored" is the target, but in 'I love being ignored by my girlfriend', it is not exactly clear if "being ignored" is a part of the target, or here, it just becomes the reason for using sarcasm against the only target : "my girlfriend". It also makes it a bit hard for our system to learn common phrases and utilize patterns for detection.
-'I love when I see people using the elevator at the gym.' The speaker could be mocking the act of people using the elevator at the gym (by saying that he/she loves to see this act), or could me mocking the people, because they use the elevator at a gym!
• Lack of Context: Context may often be necessary to determine a sarcasm target. Consider the sentence : 'Oh, you are such a lovely couple! You two could even give those Potters living down the street a real run for their money!' In this case, you/You two are being ridiculed. For 'those Potters', two scenarios are possible : a) The Potters could be a well known 'lovely couple', and their reference is used for augmenting the (sarcastic) praise or b) The Potters could be a couple with a famous negative reputation, and their reference was intended to make the listening couple aware of the speaker's sarcastic intentions. In case of the latter scenario, 'those Potters' becomes a target as well.
Another source of error is cases where the target lies outside the text. We now describe such examples and compare the impact of these errors with the overall performance. In our dataset of book snippets, there are 11 texts ( 5%) with sarcasm target outside the text. In case of tweets, such cases are much higher: 53 tweets ( 10%).

Conclusion & Future work
In this paper, we introduced a novel problem: sarcasm target identification. This problem aims to identify the target of ridicule in a sarcastic text. This target may be a subset of words in the text or a fall-back label 'Outside'. The task poses challenges such as multiple sarcasm targets or sarcasm targets that may not even be present as words in the sentence. We created and released the first dataset which labels the target of ridicule in sarcastic tweets and book snippets. To motivate automatic identification of sarcasm target, we present an introductory approach which is a hybrid of two kinds of extractors: a rule-based and a statistical extractor. An integrator then combines the outputs of the two extractors in two configurations: OR and AND. We evaluate our approach on annotated tweets as well as book snippets. In general, our hybrid OR system performs the best with a Dice score of 39.63. This is higher than two baselines: a naïve baseline designed for the task, and a baseline based on sentiment target identification. Our hybrid approach is also higher than the two extractors individually. This shows that the two extractors collectively form a good sarcasm target identification approach. We finally present an analysis of errors due to target being outside the text. The dataset and the results obtained from our introductory approach will aid sentiment analysis systems to attribute the negative sentiment expressed via sarcasm to the appropriate entity or aspect. Our work forms a foundation for future approaches to identify sarcasm targets. As future work, additional rules in the rule-based extractor and novel sets of features in the statistical extractor may be used. Using discourse relations to extract sarcasm targets will be useful. In addition, syntactic dependencies have been found to be useful in case of opinion target extraction (Qiu et al., 2011). Applying these techniques for sarcasm target identification can be useful. A special focus on the 'outside' cases (i.e., cases where the target of ridicule in a sarcastic text is beyond the words present in the sentence) is likely to be helpful for sarcasm target identification.