{'fraction_non_alphanumeric': 0.023622694219629432, 'fraction_numerical': 0.007189515632061131, 'mean_word_length': 4.99261447562777, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
MuSeM: Detecting Incongruent News Headlines using Mutual Attentive Semantic Matching

Measuring the congruence between two texts has several useful applications, such as detecting the prevalent deceptive and misleading news headlines on the web. Many works have proposed machine learning based solutions such as text similarity between the headline and body text to detect the incongruence. Text similarity based methods fail to perform well due to different inherent challenges such as relative length mismatch between the news headline and its body content and non-overlapping vocabulary. On the other hand, more recent works that use headline guided attention to learn a headline derived contextual representation of the news body also result in convoluting overall representation due to the news body's lengthiness. This paper proposes a method that uses inter-mutual attention-based semantic matching between the original and synthetically generated headlines, which utilizes the difference between all pairs of word embeddings of words involved. The paper also investigates two more variations of our method, which use concatenation and dot-products of word embeddings of the words of original and synthetic headlines. We observe that the proposed method outperforms prior arts significantly for two publicly available datasets.


Introduction
In the age of the prevalence of smart handheld devices, most of the information consumption is digital. This paradigm shift in the way people consume information has also brought forth several new challenges, such as misinformation and deceptive content. News headlines that incorrectly represent the content of the news body are called incongruent or click-baits. A deceptive and misleading news headline can result in false beliefs and wrong opinions. News titles play an essential role in making first impressions to readers and thereby deciding the viral potential of news stories within social networks [17]. Most users rely only on the news title content to determine which news items are significant enough to read [8]. The curse of deceptive content gets amplified by several magnitudes when people share it without reading news body content. Consider an example of an incongruent headline in Figure 1 taken from CNN (cnn.com). 1 The headline states that "Trump says GOP working on tax plan for middle class" whereas body content mentions that "It's unclear what tax proposal Trump was referring to on Saturday" which contradicts with the headline.
In another example in Figure 1, 2 the headline reads "The Scary New Science That Shows Milk Is Bad For You" but a part of the body text clearly states that "The study was small, to be sure, and it included no women." The headline radically generalizes the claim made in the study and exaggerates it. Many machine learning based solutions have been investigated previously in the literature to detect click-baits and news headline incongruence. Some initial works [7] [24] use sentence matching based methods to compute similarity or dissimilarity between the web claims and news headlines to detect incongruence. Researchers [2] [18] have also utilized different classification methods and used linguistic and stylistic features to learn a classifier to identify the incongruity between news titles and news body. The authors in [26] propose neural attentionbased methods to find the entailment between the news items' title and body. Some recent works [19] have identified the significance of generative methods such as generative adversarial networks for incongruence detection.
Previous methods applied to click-bait detection that use natural language processing techniques are not suitable for the detection of headline incongruence as this problem requires more facets and aspects to be covered than just stylistic features [5]. The methods that use a text similarity based approach to detect incongruence perform poorly because of the long text content of the news body. Text similarity schemes work well in case of short texts. This paper proposes a semantic matching technique based on inter-mutual attention that uses a synthetically generated headline corresponding to the news body content and original news headline to detect the incongruence. The proposed inter-mutual attention technique is inspired by some recent works, which use the intra-mutual word to word attention within a sentence to detect sarcasm [23] and intra-mutual user to user attention within a retweet propagation path sequence to detect misinformation [13].
In sum, the major contributions of this work are as follows: 1. We are the first to use inter-mutual attention based semantic matching to detect incongruent news headlines. The key idea is to get the pairwise difference between word embeddings of original and synthetic headlines and compute a mutual attention matrix after applying a dense layer. Subsequently, row-wise maxpooling can be used to compute the attention scores, to be used for the classification.
2. We use synthetic headlines generated from various generative adversarial networks based schemes using news body content to get the effective, contextual, and low dimensional representation.
3. We also investigate two additional variants of the proposed model, which incorporate addition and concatenation of word embeddings of the word pairs of original and synthetic headlines. 4. We combine all the three variants of word embedding operations in a clubbed model, which outperforms the three variants individually.

5.
We conduct experiments with two publicly available datasets, which show the effectiveness of the proposed models to detect incongruent news headlines.

Related Works
Most of the literature's initial works propose using linguistic and statistical features based classifiers to detect click-baits and incongruent headlines. The authors of [4] suggest to perform lexical and syntactic analysis for the identification of the clickbaits. In [3], authors use linguistic features to learn support vector machine (SVM) based classifier to detect click-baits, and they also release a manually annotated dataset. Authors of [16] use text features and meta-information of tweets to learn a classifier to detect click-baits.
On the other hand, some works [7][24] deal with sentence matching based stance classification which is a closely related problem to headline incongruence. These methods are not suitable directly for headline incongruence due to some challenges such as relative length mismatch between the news headline and its body content and non-overlapping vocabulary. Authors in [25] propose to use a co-training approach with myriad kinds of features such as, e.g., sentiments, textual, and informality. Some very recent works such as [26] use neural attention [1] based approach to achieve headline guided contextual representation of the news bod text. This headline guided attention also results in convoluting the body content's overall representation due to its lengthiness.
We propose an improved semantic matching between the news headline and its body text via inter-mutual attention. The intermutual attention based matching is performed between the original and a synthetically generated headline, rather than between original news headline and its body content. The synthetic headline is generated via state-of-the-art generative adversarial methods using the body contents of the news item. The generative adversarial techniques have been very successful in generating realistic artificial images [12], videos [20] [14] and text [10] contents. One of the pioneer work in artificial text generation is [27], which solves the problem of generator differentiation by updating the gradient policy directly. The authors of [10] introduce a GAN based text generation method, which uses a third module, called MANAGER, which helps the generator network to utilize some leaked feature information from the discriminator network. In [19], authors introduce a synthetic headline generation method based on style transfer and generative adversarial network.
The key idea behind the inter-mutual attention method is inspired by two recent contributions [23] [13]. There are some significant differences between these works and the proposed model. Firstly, they use intra-mutual word to word or user to user attention, which builds on word embedding pairs or user embedding pairs occurring within a sentence or retweet path sequence correspondingly to compute the attention scores. In contrast, the proposed model uses pairs of word embeddings of words that belong to two different sentences, i.e., original and synthetic headline. Secondly, these previous methods use a concatenation of embedding vectors to form an overall pair embedding representation. In contrast, the proposed method uses the difference between the word embeddings to form an overall pair embedding representation. The intuition behind these differences is evident as we detect incongruence between two pieces of texts (headline and body), and computing the difference between word embedding pairs results in capturing the  cues related to semantic similarity or dissimilarity between the two.

Problem Definition and Proposed Model
This section introduces the problem definition and presents the overall architecture of the proposed model MuSeM in detail. We describe the synthetic headline generation schemes, and then we provide details of the inter-attentive semantic matching technique. We also present two more variants of MuSeM model and devise a clubbed model, which combines all the three variants.

Problem Definition
Given a news item n i ∈ N, having headline h i and body content b i , we need to predict whether there is incongruence between h i and b i by classifying the news as either "Congruent(C)" or "Incongreunt(I)." The news headline h i consists of a sequence of l words denoted as h i = {w h1 , w h2 , ..., w hl } ∈ W, and the news body content b i consists of a sequence of m words denoted as

Word Embedding Layer
For a news item n i , the corresponding headline h i of length l and body content b i of length m are represented as word embedding vector of dimension d for the k th word in body content b i . We experiment with pre-trained glove embeddings for the evaluation.

Synthetic Headline Generation
As discussed in the introduction, the semantic similarity between the headline and news body content is the most significant incongruence indicator. Text similarity schemes work well in case of short texts. However, the major bottleneck to compute such a similarity measure is the news body's text length. For this reason, the methods which use a text similarity based approach to detect incongruence perform poorly. Generative adversarial networks (GANs) are recently getting much traction for various application scenarios, from the generation of realistic artificial images to compositions of meaningful poems. We utilize GANs based text generation techniques to generate Preprint -MuSeM: Detecting Incongruent News Headlines using Mutual Attentive Semantic Matching 4 a short synthetic headline for each of the news items using respective news body contents. The resulting synthetic headline is a low dimension contextual representation of the news body content.

Generative Adversarial Network
A typical generative adversarial network [9] comprises two neural nets, a discriminator and a generator. Both of these sub neural nets compete with each other. The generator network tries to maximize the classification error, while the discriminator network tries to minimize it. Both the generator and discriminator converge together and reach an equilibrium state.

Sequence Generative Adversarial Nets (SeqGAN)
SeqGAN [27] is a sequence generation method based on generative adversarial networks. The key difference between standard GANs and SeqGAN is that SeqGAN bypasses the generator differentiation problem associated with original GANs by directly performing a gradient policy update using a reinforcement learning (RL) based approach. In the θ-parameterized generative model G θ and start state s 0 , the expected end reward for output y 1 is defined as: where Q G θ D φ (s 0 , y 1 ) is an action-value function. A recurrent neural network (RNN) is used for the generative model for sequences, and a convolutional neural network (CNN) is used as the discriminator. We refer to the respective paper for more details.

Stylized Headline Generation (SHG)
Since incongruent headlines and click-baits usually follow a certain catchy and deceptive writing style, it can be advantageous to generate synthetic headlines that mimic these similar writing styles. Authors of very recent work SHG [19] propose a style transfer based headline generation approach, which uses a generative adversarial network with style discriminator. A gated recurrent unit (GRU) based recurrent neural network (RNN) is used as a generator that tries to minimize the following negative log-likelihood: (2) Three variants of discriminators are used, one for distinguishing the styles of the original and generated headline, a second one to maintain the aligned distributions in both the original and generated headlines, and a third for making sure that headline and body pairs are correctly classified. Again, we refer to the respective paper for more details.

Inter-mutual Attentive Semantic Matching
We now discuss the proposed semantic sentence matching scheme in detail. Our key idea is to model the relationships between all possible pairs of the words occurring in both the sentences, which captures the inherent semantic similarity between the sentences. We use pretrained word embeddings vectors and apply a novel inter mutual attention technique to model the similarity or dissimilarity relationship between pair of sentences.

Word to Word Inter Mutual Attention
We compute mutual attention scores between a pair of sentences, of which first is the original headline of the news item, represented in terms of a sequence of word embeddings, and second is the synthetically generated headline for news item n i , represented in terms of a sequence of word embeddings. Here, l and p are lengths of the original headline h o i and the synthetic headline h s i , respectively. First of all, we compute the difference between word embedding vectors of each candidate word pair W q , W r . Candidate pairs are formed by selecting all possible combinations of the inter sentence word pairs, such as W q , W r , where W q and W r are the q th word of the original headline h o i and the r th word of the synthetic headline h s i , respectively. Now we use a dense layer to project the difference of candidate embedding pairs into a scalar score: where θ diff ∈ R d×1 is a weight matrix and b diff ∈ R a bias term.
The score matrix C = (C qr ) is of dimension l × p. We use rowwise avg-pooling and apply softmax to compute inter mutual attention scores for the original headline: where A o is the learned inter-mutual attention weight vector for original headline. We use column-wise avg-pooling and apply softmax to compute inter mutual attention scores for synthetic headline.
where A s is the learned inter-mutual attention weight vector for synthetic headline. Subsequently, inter-mutual attended representations for original headline M A o and for synthetic headline M A s can be computed as: where f (w h o i ) and f (W h s i ) are the word embeddings of the i th word of the original and synthetic headline, respectively. Now, we compute the overall inter-mutual attended representation M A as:

Variants of Inter Mutual Attention
Although the MuSeM model with difference of word embeddings of candidate word pair works well, we also investigate and experiment with two other versions of MuSeM that use the dot product of candidate word embedding pairs and concatenation of word embedding pairs, similar to [22]. We notice that the dot-product and concatenation variants do not perform better than the difference-oriented variant in our experiments. We also combine all three operations, namely difference, dotproduct, and concatenation, which performs slightly better than Preprint -MuSeM: Detecting Incongruent News Headlines using Mutual Attentive Semantic Matching 5 the purely difference oriented model. Attention scores for the dot-product and concatenation oriented models can be computed in a very similar fashion to the difference model, except for equation (3), which needs to be replaced by: For the sake of brevity, we do not repeat all the other equations.

Clubbed Model
We also experiment with a combined model in which all three variants namely difference based, dot-product based and concatenation based methods are used in parallel and resulted overall embedding representation is used for computing attention scores.
where C qr is overall attention score, θ d pc is the weight matrix and b d pc is the bias term for the clubbed model.

LSTM based Sequence Encoder
We concatenate the sequence of word embeddings of the words of the original headlines and words of the synthetically generated headline and use long short-term memory unit (LSTM) [11] to encode this overall sequence by using the standard LSTM equations as follows: where h t−1 is previous hidden state and x t is the current input. We use the last hidden state of LSTM as the encoded representation of the overall sequence M E .

Classification
We learn a joint representation M of inter-mutual attended representation M A and encoded representation of the overall sequence M E using a nonlinear transformation layer with ReLU activation. Subsequently, we use a softmax layer to predict the label: where W cl , b cl , andŷ are the weight matrix, the bias term, and the predicted label, respectively. We use softmax cross-entropy with logits as loss L:

Experimental Setup
We implement the proposed models within the TensorFlow framework. For the evaluation and comparison, we use Macro F1 and AUC scores as evaluation metrics. All the parameters are tuned using a grid search. As a result, we keep learning rate as 0.001, batch size as 100, hidden states of LSTM as 100, and pretrained GloVe [15] embeddings of 300 dimensions. We use softmax cross-entropy with logits as the loss function, maximum sentence length as 50, dropout rate as 0.2, and the number of epochs as 10. For the synthetic headline generation, we use the default parameter values used in the papers mentioned above.

Dataset Statistics
We use two publicly available datasets, NELA17 3 and Clickbait Challenge 4 for the evaluation and comparison of the proposed model with the baseline methods. The NELA17 dataset is provided by [26]. Although they do not evaluate their model with it, they provide a script 5 to generate the dataset from an original news collection dataset. It contains 91042 news items in total, of which 45521 news items are congruent, and 45521 news items are non-congruent. The Click-bait Challenge dataset is a collection of social media posts, which are annotated as click-bait or non-click-bait using a crowd-sourcing platform via majority voting. It contains 21033 social media posts in total, of which 16150 posts are congruent, and 4883 posts are non-congruent. The NELA17 dataset is balanced but the Click-bait Challenge dataset has class imbalance problem. We use the class weight 6 parameter to adjust the imbalance in the dataset.

Evaluation and Discussion
In this section, we compare the performance of the proposed models with state of the art methods and baseline models on two publicly available datasets. Subsequently, we also provide some explanations on the differences in the performance of the analyzed models. We also discuss a potentially useful trick from [13], which can improve the incongruence detection accuracy by capturing the newly uncovered cues.
We compare the proposed model with these baselines: • SVM: [6] In this method, we use handcrafted linguistic and other statistical features to learn a classifier with support vector machine technique.
• LSTM: [11] In this method, we use long short term memory unit to encode both headline and body pair and apply softmax for the classification.
• Hi-LSTM: [21] This technique uses an LSTM based hierarchical encoder, which first encodes words and then uses another LSTM encoder to form the sentence representations.
• Yoon: [26] This is a state of the art, hierarchical dual encoder based model which uses headline guided attention to learn the contextual representation.
We experiment with four variants of the proposed MuSeM model.
• MuSeM diff SeqGAN: This is the MuSeM model with the difference between word embeddings, and synthetic headlines are generated using SeqGAN method.
• MuSeM dpc SeqGAN: This is the MuSeM model which uses a combination of all three operators, namely difference, concatenation, and dot-product be-tween word embeddings and synthetic headlines are generated using SeqGAN method. • MuSeM diff SHG:This is the MuSeM model with the difference between word embeddings, and synthetic headlines are generated using SHG method. • MuSeM dpc SHG: This is the MuSeM model which uses a combination of all three operators, namely difference, concatenation, and dot-product between word embeddings and synthetic headlines are generated using SHG method.

Results for Click-bait Challenge Dataset
In the case of the Click-bait Challenge dataset, we observe very similar trends as with the NELA17 dataset. The deep learning based methods perform significantly better than the handcrafted feature based SVM model. With 0.660 and 0.678 in Macro F1 and AUC, the Yoon model outperforms both the LSTM and Hi-LSTM models. All variants of the MuSeM model perform better than all the baseline methods, whereas MuSeM dpc SHG model achieves 0.735 and 0.747 in terms of Macro F1 and AUC, beating all the other variants.

Higher Order Inter-mutual Attention
Authors in [13] propose a variant of mutual attention which can be used to model higher-order relationship among the candidate words. Essentially, the proposed inter-mutual attention only models the relationship between two words at a time individually, and the presence of other words in the sentences are not taken into account. If we can also reckon the presence of other words during the computation of the inter-mutual attention, we can capture the additional contextual cues related to similarity or dissimilarity relationship. The higher-order mutual attention trick allows us to model such relationships.
There are certain challenges to be taken care of before the application of higher order mutual attention to incongruence detection problem. Firstly, the original higher order mutual attention is designed for a single sequence as it uses intra-mutual attention within a sequence, while headline incongruence detection involves two sentences or sequences. Secondly, higher order mutual attention is achieved by applying a square or cube of the attention score matrix, which is possible only if the original score matrix is a square matrix. In the headline incongruence detection task, the score matrix may or may not be a square matrix because original and synthetically generated headlines may have different lengths.

Conclusion
This paper proposes inter-mutual attention based semantic matching (MuSeM) to detect incongruence in news headlines. We also investigate a different variant of MuSem, which combines three operations on word embedding pairs to compute inter-mutual attention scores. The proposed models outperform all the baselines in experiments with two publicly available datasets. We notice that the performance of inter-mutual attention based semantic matching greatly depends on the accuracy of synthetic headline generation step.
In future research, we plan to use the higher order word to word attention trick used in [13] to model and capture the indirect relationships between the word pairs. We plan to investigate and devise an end to end version of MuSeM model, where synthetic headline generation and semantic matching steps are seamlessly integrated. We also plan to conduct an analysis of Preprint -MuSeM: Detecting Incongruent News Headlines using Mutual Attentive Semantic Matching 8 the efficacy of the attention mechanism by visualizing the attention weights.