{'fraction_non_alphanumeric': 0.03164887890997447, 'fraction_numerical': 0.012702611263187034, 'mean_word_length': 4.767961681745609, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Prediction of People’s Emotional Response towards Multi-modal News

We aim to develop methods for understanding how multimedia news exposure can affect people’s emotional responses, and we especially focus on news content related to gun violence, a very important yet polarizing issue in the U.S. We created the dataset NEmo+ by significantly extending the U.S. gun violence news-to-emotions dataset, BU-NEmo, from 320 to 1,297 news headline and lead image pairings and collecting 38,910 annotations in a large crowdsourcing experiment. In curating the NEmo+ dataset, we developed methods to identify news items that will trigger similar versus divergent emotional responses. For news items that trigger similar emotional responses, we compiled them into the NEmo+-Consensus dataset. We benchmark models on this dataset that predict a person’s dominant emotional response toward the target news item (single-label prediction). On the full NEmo+ dataset, containing news items that would lead to both differing and similar emotional responses, we also benchmark models for the novel task of predicting the distribution of evoked emotional responses in humans when presented with multi-modal news content. Our single-label and multi-label prediction models outperform baselines by large margins across several metrics.


Introduction
Understanding how exposure to certain textual and visual news affects people's emotional reactions is important for detecting, educating, and correcting intentional or unintentional emotional manipulation of readers. As a step towards detecting such manipulations and raising news consumers' visual literacy, in this work we develop methods for predicting emotional responses towards news headlines and images. To the best of our knowledge, machine learning tools that predict how a reader will react emotionally to a certain news headline, choice of a lead image, or combination of both do not exist. In this paper, we introduce tools that enable such prediction and thus can shed light on effects of news presentation, which is important to both editors and consumers of news.
The dataset we utilize in this work has been developed in phases. It first started with the headlines of news articles in the Gun Violence Frame Corpus (GVFC) (Liu et al., 2019), along with corresponding lead images of these articles (Tourni et al., 2021). A previous study started a crowd-sourcing experiment to collect emotional response annotations to the news headlines and images, producing the BU-NEmo dataset (Reardon et al., 2022). In this work, we extend the above emotional response experiment significantly. We utilize our new expanded dataset, named NEmo + , and present the first benchmark of models to predict the evoked emotional responses in news consumers when presented with multi-modal news content.
2 Related Works 2.1 Predicting Emotional Responses to Text Sentiment analysis is the task of detecting positive vs. negative sentiment expressed by text. The previous works on text-based emotion prediction have mostly focused on binary classification of positive versus negative emotions (Jiang et al., 2011;Wang et al., 2018). In our work, we aim to predict which category of emotions, from multiple choices, a text will elicit, a task for which there is limited prior work. Ahmad et al. (2020) focus on multi-class emotion state classification in poetry and Vasava et al. (2022) aimed to predict the type of emotion in essays written in response to newspaper articles. While Vasava et al. (2022) classified each essay into one of six basic emotions (Ekman and Friesen, 1971), we use the eight emotions from the prominent psychological study by Mikels et al. (2005) as our categories. The major difference between our work and that of Vasava et al. (2022) is that the essays used in their study already con-tain readers' sentiments on the newspaper articles. We present the novel task of directly predicting the emotional reactions of readers to news headline text, without such essays. The recent study of Gabriel et al. (2022) involves modeling how readers react to news headlines. Their work however, focuses on free-text explanations of readers' reactions and ordinal estimates of likelihood of spread and identification of real vs fake news headlines. By contrast, our dataset contains categorical emotional labels in order to predict the emotional responses. Gambino and Calvo (2019)'s study is closely relevant to ours as they also focused on the novel task of predicting the evoked emotion rather than the previous research of identifying the presence or absence of an emotion. They collected a group of news articles and their associated tweet responses and annotated the emotions expressed in them. They are predicting the evoked emotions towards the whole news article and we are using only the headline as we aim to explore how specific choices of the headline text by the news editors affect the emotion reactions.

Predicting Emotional Responses to Images
Recent computer vision work has focused on building models to recognize the emotional state of specific persons in images (Li et al., 2021;Zhang and Xu, 2022), rather than the emotional state that images can elicit in humans. There is very limited work on predicting these reactions to visual data (Machajdik and Hanbury, 2010;You et al., 2016;Achlioptas et al., 2021). The most relevant of these works is the ArtEmis dataset (Achlioptas et al., 2021), which contains more than 80k art-related images with annotations of (1) emotional reactions of crowdworkers towards images and (2) their freeflowing English textual explanations of how and why they felt a certain way. Studies with ArtEmis predict (1) by analyzing (2), a task far simpler than ours since their model input is an explanation of an emotion that the model then learns to extract. In our task, emotional reactions must be predicted from the original news headlines and images.

Predicting Emotional Responses to
Multi-modal Content Multi-modal models have gained success in predicting and understanding emotions by combining audio, textual, and visual data (Busso et al., 2008;Poria et al., 2019;Dudzik et al., 2020). Most of the previous multi-modal models for emotional pre- diction focused on combining elements that are homologous in nature. For example, the MELD dataset (Poria et al., 2019) predicts emotions using multiple modalities (audio, textual, and visual), which were all part of the same video source. The BU-NEmo dataset created by Reardon et al. (2022) is novel in that the modalities (news headline and image) were separate in nature and chosen to be presented together by the news publishers. We significantly extended this dataset to create the NEmo + dataset in order to have enough training data for multi-modal models. Multi-modal learning on the NEmo + dataset can give us an idea of the likely emotional reactions evoked by a specific combination of inputs from multiple modalities (news headlines and images). There are limited datasets available for pretraining our models for both text-to-emotion and image-to-emotion prediction. Most of the datasets mentioned above use different sets of emotional labels than in our NEmo + dataset. ArtEmis (Achlioptas et al., 2021) provides the same 8 emotional labels as ours in addition to a 9th "something else," so we used ArtEmis to pre-train some of our models.

Dataset Collection
BU-NEmo (Reardon et al., 2022) previously extended the work of the Gun Violence Frame Corpus (GVFC) (Liu et al., 2019; which applied frame detection on gun violence related news headlines, and created 1,300 news headline and image pairings. Reardon et al. (2022) initially annotated the news items in GVFC with emo-tional responses by workers from Amazon Mechanical Turk (MTurk) with annotators of at least high school qualification. A significant portion of the annotations contained spam in the free flow written text making the quality of the categorical emotional responses questionable. This spamming on MTurk is consistent with other findings of MTurk's low annotation quality (Rashtchian et al., 2010). Due to this limitation, we decided to implement a survey website (hosted on AWS) with the same survey content and interface to the study of Reardon et al. (2022) to collect the annotations for this study. We awarded course credits to anonymous student participants from the College of Communication and the Computer Science Department at Boston University through an internal annotation collection system managed by the university. We received high quality responses.
For our data collection, we followed the same pipeline as the BU-NEmo study (Reardon et al., 2022). Our pool of annotators consisted of undergraduate and graduate university students. The BU-NEmo dataset contained 320 news items with 10,547 annotations. Our NEmo + dataset is significantly expanded, by adding 977 news items and 28,363 annotations to the original dataset. For each news sample, there are three experimental conditions: presenting only the headline text to the annotator (condition T), only the lead image (condition I), or the headline and image together (condition TI). For each experimental condition, we obtained 10 annotations per sample with each providing: the dominant emotion that the annotator feels among eight emotional categories (Amusement, Awe, Contentment, Excitement, Fear, Sadness, Anger, and Disgust), the intensity of the emotion on a scale of 1-5, and a free-flow English written text describing why the annotator feels that emotion. The overall distributions of responses across the eight emotions in NEmo + are shown in Figure 1.

Prediction Difficulties in NEmo +
We identified some interesting properties in the dataset that make it challenging to predict a single emotional response, which we discuss in detail below. These are intrinsic to the nature of the dataset and are not limitations of the machine learning models benchmarked in this study.

Limited Context Carried in Images
Some of the news images or headlines do not carry much context on their own, like the example shown in Figure 2. This image provides no clear indication of the identity of the person in the image nor the content of her speech, while the corresponding headline gives more context into the original news content. As we can observe from the viewers' freeflow responses when presented with only the image (I condition), their reported dominant emotions depend largely on speculations. The sample image elicits no negative emotions like sadness, which is present in both the T and TI conditions. In such news items, the headline text is essential in helping viewers form holistic emotional impressions.

Emotional Diversity
Another interesting property we observed is that many news items evoke a diverse set of emotional reactions. In the example in Figure 3, annotators have differing emotional reactions towards a given news sample, when presented with the image and headline separately or together. Even positive emotions (Excitement, Awe, Contentment, Amusement) can vary significantly as shown in the example. Moreover, as can be observed from the T condition, while written responses suggest annotators agree in a sense, some viewers express negative emotions like anger instead of positive emotions, as they feel that the younger generation should not have to fight for safety.

Dataset Curation
For the rest of the discussion, let n labels be the number of emotional response types that serve as labels for a news sample and m the number of people that annotate each news sample. We define v ∈ N n labels to be the frequency annotation vector of a sample, and the entry v i ∈ {0, . . . , m} describes how many annotators experienced the emotion expressed by the i-th label. To curate the NEmo + dataset for our purposes, we process n labels = 8 possible emotional responses (amusement, awe, contentment, excitement, fear, sadness, anger, disgust); in this order, of m = 10 experiment participants. A frequency annotation vector of (0, 0, 1, 0, 0, 2, 0, 7), for example, means that 7 participants experienced the emotion 'disgust', 2 the emotion 'sadness', and one the emotion 'contentment'.
In Section 3.2, we observed that the 1,297 news data points of the NEmo + dataset elicited two types of responses: (1) noticeable emotional consensus in the annotations and (2) varying emotional responses with no clear inclination towards a single emotion. We design a subset of the NEmo + dataset, the NEmo + -Consensus ("NEmo + -C") dataset, that only includes news item with emotional consensus, removing those samples for which people had varying opinions. For this, we experimented with two different filtering methods, discussed below.

Filtering by Rank Diff: Nemo + -CR
We defined the rank difference for a news sample to be the difference in frequency between the most frequent emotional response by the group of annotators and the second most frequent emotional response by the group. For the example frequency annotation vector described above, (0, 0, 1, 0, 0, 2, 0, 7), we sort the entries to yield {disgust: 7,  sadness: 2, contentment: 1}. Then the rank difference is the frequency difference between the highest ranked emotion 'disgust' and the second highest emotion 'sadness,' which is 5. This approach is similar to the margin of confidence uncertainty used by Scheffer et al. (2001) as it also examines the difference between the highest and second highest items. In the rank filtering method, we process the NEmo + dataset to only keep news items that have a rank difference of greater than or equal to τ rank . Any news sample, for which the rank difference of the frequency annotation vector lower than τ rank , is removed. We call this filtered dataset Nemo + -CR for "Consensus by Rank." We chose τ rank = 3 to balance having enough consensus in the total m = 10 annotations for a particular news sample and having enough data for training machine learning models. The size of Nemo + -CR for τ rank = 3 is shown in Table 1.

Filtering by Entropy: Nemo + -CE
The frequency annotation vector can be considered a probability distribution of emotions. If the participants' emotional responses vary strongly for a news sample, we consider the response uncertain. If there is consensus among the participants, however, we consider the response certain. Since entropy is a measure of the uncertainty of a probability distribution, we can use it to filter the news items. We keep those news items with small entropy values, containing less uncertainty in the emotional distribution of the frequency annotation vector. This is similar in spirit to the rank difference filtering as both methods aim to select those news items that evoke strong emotional consensus. For a fair comparison, we selected the entropy filtering threshold so that the resulting filtered dataset is similar in size to the rank difference-filtered dataset. The size of the resulting filtered dataset Nemo + -CE (Consensus by Entropy) is shown in Table 1.

Method
We benchmark machine learning models, described in detail in Section 5, for each of the three conditions (T, I, TI) to examine whether text or image when presented separately or together provide more context and help viewers form an emotional response towards particular news content.

Prediction on the Consensus Data
We performed single label-classification on NEmo + -Consensus (Nemo + -CR and Nemo + -CE).
As each news sample has m = 10 emotional annotations, we first need to create the single ground truth representative emotion for each news sample.
The single representative emotion we use for prediction in the following discussions is simply the most frequent emotion in the m annotations.

Classification on Headline Text
For the T condition, our system aims to predict the single emotional label based on the headline text as the input. This becomes an n labels -class classification task.

Classification on News Image
For the I condition, we developed two separate approaches. The first approach, intuitively, is to predict the emotional label based on the image data itself. However, due to the limited size of the Nemo + -CR and Nemo + -CE datasets, it is difficult for our system to extract meaningful features from 2-dimensional image data. Furthermore, the images in our dataset do not always provide enough context to the actual content of the news as discussed in Section 3.2.1. In order to infuse some context into image data, we mapped images to text using the Google Web Entity Tagger API 1 that uses pre-trained models 1 https://cloud.google.com/vision/docs/detecting-web to quickly assign web entity tags and labels to our images (see Figure 4). These tags include textual context of the news that are not always available in the raw images. We also used another image-totext conversion approach based on the automatic image captioning method by Tourni et al. (2021). After converting the images into textual data, we used the same pipeline as for text classification.

Classification on Image+Text
For condition TI where we are predicting the emotional response of the annotators when presented with both the headline text and the image, we used a multi-modal classification approach where the model learns from both the headline text and the news image to predict the emotional reactions.

Prediction on the Full NEmo + Data
One limitation of the single-label classification is the reduced dataset from the filtering of the dataset in order to select the dominant "consensus" emotion. The filtering methods mentioned above (rank difference and entropy filtering) aim to select news items that have strong emotional consensus and have a clear dominant emotion. However, most of the time people expressed diverse emotions. In fact, more than 60% of the data in all three conditions in our NEmo + fall into this category of having no clear consensus, as shown by the sizes of the filtered datasets in Table 1 (NEmo + contains 1,297 news items in total). Our approach to the dilemma of having limited consensus in our dataset is multi-label classification. For every news sample, we turned the frequency annotation vector v from the 10 annotations into a list of binary labels based on a fixed frequency threshold t. We set each entry v i ∈ {0, . . . , m} of the frequency annotation vector v to 1 if v i ≥ t and zero otherwise. For example, for a frequency threshold t of 2, we turned the frequency annotation vector [0, 0, 1, 0, 1, 2, 1, 6] into [0, 0, 0, 0, 0, 1, 0, 1].

Text Models
Due to the recent success of Bidirectional Encoder Representations from Transformers (BERT) in the text classification task (González-Carvajal and Garrido-Merchán, 2020), we used BERT (Devlin et al., 2019) for the text classification machine learning models on our emotional consensus dataset. We also experimented with RoBERTa and observed similar results, so we chose to use the smaller, more efficient BERT model as the main text classification model for news headlines, image tags, and image captions.
Since our dataset is relatively small for training a deep neural network from the ground up, we explored the approach of whether learning from a related domain will be helpful. The ArtEmis dataset provides a foundation for training our baseline models as Achlioptas et al. (2021) used the same eight emotions as Mikels et al. (2005), in addition to a ninth emotion "something else." We removed all records containing the emotion "something else" in the ArtEmis dataset and used the remaining 401,722 data points to train a text-toemotion baseline BERT model, and then fine tuned it with our consensus data: NEmo + -Consensus (Nemo + -CR and Nemo + -CE). We refer to this model as A-BERT. We also directly fine tuned a BERT-base-uncased 2 model without pre-training with ArtEmis data for comparison 3 .

Image Models
For predicting the emotional response on solely the image data in NEmo + -Consensus, we followed the pipeline of the ArtEmis study (Achlioptas et al., 2021) and used a Resnet34 architecture with initial weights that have been pre-trained on the ImageNet dataset with 100,000+ images (Deng et al., 2009) and used the KL-divergence of the frequency annotation vector (from the annotations in the I condition) relative to the network output (normalized to a probability distribution) as the loss function.
The output of the model is a distribution of the likelihood of each emotion. We compared the maximum likelihood predicted emotion with the most frequent emotion in the ground truth to measure the performance of the single label prediction. We then fine tuned on the NEmo + -Consensus dataset and refer to this model as A-ResNet. We also directly fined tuned an imageNet based Resnet model without pre-training with ArtEmis for comparison.

Multimodal Image and Text Model
For predicting the emotional response when the viewers are presented with both the headline text and the image in NEmo + -Consensus (NEmo + -CR and NEmo + -CE), we fine tuned a BERT based multi-modal bitransformer model introduced by Kiela et al. (2019) using both the headlines and images.
We did not pre-train the multimodal models with ArtEmis because unlike NEmo + , there is no single text (i.e., headline) for every image in ArtEmis. Instead, for each image in ArtEmis, there are multiple free flow text responses indicating various emotions. It is not straightforward to choose the "best" text to pair with an image for an indicated emotion as some free flow responses might be better at indicating emotions than others. We leave such exploration for future work.

Models for NEmo + with Diverse Emotions
Since the NEmo + dataset contains news data points where there is no emotional consensus, we performed multi-label text classification by fine tuning BERT for all three conditions. For condition I, we converted the image to textual data using the Google Web Entity Tagger API. For condition TI, we concatenated the tagger converted text with the original news headline text as the input to the multilabel model.

Single-label Classification
The main metric we used for single-label classification is accuracy in predicting the most frequently elicited emotion. Since there are n labels = 8 classes, the expected classification accuracy based on a random guess, i.e., picking a class uniformly at random among all classes (independently for each sample) is given by 1 /n labels = 12.5%, a rudimentary baseline for accuracy. However, the NEmo + -Consensus dataset is imbalanced towards negative emotions. Therefore, we also compared our models to the majority baselines (the percentage of the dominant emotion in the dataset) to take into account the imbalanced nature of the dataset. These are shown in Table 2. As can be seen in Table 2, rank difference filtering (Nemo + -CR) provides a more consistent sample size with emotional consensus across all 3 conditions than entropy filtering.

Multi-label Classification
For multi-label prediction, we used Hamming distance (Sorower, 2010), exact match accuracy, and  rank-based average precision (LRAP) 4 to evaluate each model's predictions.

Results
We split the datasets into train / validation / test sets in the ratio of 50%:25%:25% and all of experiment results are reported on the test set.

Single-label Prediction on Consensus Data
The test time performance of all of our single label prediction models on the data with emotional consensus (Nemo + -CR and Nemo + -CE) is shown in Table 3. BERT and A-BERT refer to the models with and without pre-training with the Artemis textual data as described in Section 5.1. ResNet and A-ResNet refer to the models with and without pre-training with the Artemis image data described in Section 5.2. As shown in Table 3, all of the models we benchmarked outperform the majority baselines in Table 2. Our best model (A-BERT on Nemo + -CR) surpasses the random baseline significantly by more than 55 percent-points and the majority baseline by 26 percent-points for the I condition. When only headlines are used (T condition) transfer learning from the ArtEmis textual data improves the accuracy in both consensus datasets. However, when only images are used (I condition), transfer learning from the ArtEmis image data improves accuracy only when images in Nemo + are converted to text. This may be due to intrinsic differences between ArtEmis and NEmo + images. Unlike artcentric images of ArtEmis that can intrinsically convey emotional meaning by themselves, images used in news articles may require additional context in the form of web-tagging or image-captioning to leave similar emotional impressions.
We observe that for the single-label prediction task, all the image-only models outperform text-  Table 3: Classification accuracies of predicting a person's emotional response on each filtered dataset for all single-label models. The accuracy of the random guessing benchmark is 12.5% and the majority baselines for each condition is shown in Table 2. I-Tag and I-Caption refer to models where the image data was converted into text using either the Google Web Entity Tagger API or the GVFC's automatic captioning. All results from this table are from the mode across 30 runs.
only models as well as models for text combined with image in both filtered datasets. Moreover, Table 1 shows that there are more samples with above-threshold consensus for the I condition than for the T or TI conditions. From this, we hypothesize that lead images may be more likely to evoke similar and more-predictable emotional responses in multi-modal gun violence news. Somewhat surprisingly, the combined text with image TI models have the worst performance in both datasets and we discuss possible reasons for this in Section 8.

Multi-Label Prediction on NEmo +
In our multi-label experiment, we controlled for the frequency threshold we used to convert the frequency annotation vectors into binary labels. The higher we set the frequency threshold to be, the easier the task would become, as the converted binary labels would be more sparse and the emotional distribution would be more concentrated.
For multi-label prediction, we are interested in data points with at least two positive binary labels. As shown in Table 4, the percentage of the training data with at least two positive labels decreases as we increase the frequency threshold for binary conversion. We observe that after a threshold of 3, the multi-label learning task becomes insignificant as the training data contains too few qualifying samples. Therefore, we focus on the frequency thresholds of 1, 2, and 3.
We simulated the random baselines by randomly choosing one of the n labels = 8 emotions m = 10 times for each of the 1,297 news items and converting the random frequency annotation vector into a  Table 4: Percentage of the data points that contain at least two 1's after the conversion to binary labels using different frequency thresholds.
list of binary labels given a fixed frequency threshold, as described in Section 4.2. We then compared the random binary labels to the actual binary labels to compute the random baselines' multi-label performances. As shown in Table 5, our models consistently outperform the simulated random benchmark in every condition (T, I, TI), at every threshold (1, 2, 3), and under every metric (higher exact match accuracy and LRAP scores and lower Hamming distance loss). Moreover, for every condition and every metric, the absolute performance improvement of our models over the random benchmark increases with threshold value and attains the highest improvement at threshold 3.  At threshold 3, compared to the random baseline, our model's Hamming distance loss is lower by 0.22, 0.24, and 0.23 points, exact match accuracy is higher by 18, 28, and 20 percent points, and LRAP score is higher by 0.25, 0.27, and 0.22 points, for the T, I, and TI conditions, respectively. In terms of absolute performance, with increasing threshold the Hamming distance and exact match metrics for T, I, and TI improve, but the LRAP metric becomes worse. As the threshold increases there are fewer examples with many labels (see Table 4). A smaller label space makes the classification task "simpler," but with fewer examples it becomes harder to generalize. Hamming distance and exact match seem to gain more from a reduced label space than they loose due to reduced sample size. The reverse seems to occur for LRAP.

Limitations & Future Work
There exist some limitations to our work. Firstly, the multi-modal classification model we benchmark in the TI condition has exhibited lower performance than in the T and I condition (Table 3). This aligns with findings of Wang et al. (2019) that different modalities generalize and fit at different rates and are prone to overfitting due to increased capacity. We also attribute the lower multi-modal prediction performance in the TI condition to the limited size of NEmo + -Consensus. It is more difficult for the model to learn enough features from multiple modalities with that amount of data.
One limitation with our multi-label experiment is that the conversion to binary labels causes a loss in relative scale of information among the n labels emotional categories. An alternative approach to this problem in future work could be to model the distribution of emotions for each news sample with a KL-Divergence loss instead.
In future work, we could also derive deeper insights by using the intensity scores we collected in Section 3.1 to predict the strength of emotional responses to news. Another future task is to predict whether a given news headline and/or image will elicit emotional consensus, or result in a divided response among readers. It will also be interesting to study the relationship between emotional responses and the framing of the news and to extend the task to multilingual setting (Akyürek et al., 2020). Finally, we are interested in making the benchmarked systems for predicting emotional responses to news accessible to researchers from a diverse array of disciplines (in similar fashion to the interactive computational framing website: Open-Framing (Bhatia et al., 2021;) so that researchers from various disciplines can conduct further studies on the potential benefits and 372 risks of such system.

Conclusion
We have shown that we can effectively, to some degree, predict the emotional response to news headline and image using standard text-and vision-classification models. Our work is the novel attempt at benchmarking the task of predicting how exposure to certain textual and visual news affects people's emotional reactions. This task has wide implications for both news consumers and news professionals. Potential misuses are the possibilities that our tool can be intentionally used to predict and manipulate the emotional reactions of news consumers with specific choices of news headlines and images. However, news editors could aim to avoid sensationalizing their produced media content by using prediction systems like ours. This would be useful in situations where presentation of sensitive news topics (war crimes, terror, etc.) benefits from a more informed selection of image-to-text combinations that can convey important information over sensational, distracting content. Publishers and experts can use this tool to recognize and avoid emotionally-manipulative content. Social media platforms could also use insights on evoked emotion from media in order to predict whether a post is likely to be click-bait. Educators could also use our system for teaching visual media literacy.

Ethical Considerations
Our NEmo + is crowdsourced from students through a U.S.-based university in the Northeast. Our dataset may contain certain political and sociocultural perspective skews given the narrow demographic. As we expand our dataset, we will incorporate annotators from diverse backgrounds while maintaining the annotation quality. We acknowledge that we have received permission to use the BU-NEmo dataset (Reardon et al., 2022), as their data is freely available for the purpose of academic research in our study. Regarding our annotation collection, we ensure we are not knowingly introducing bias to the data nor inflicting any emotional harm on participants or breaching their confidentiality, for which we have obtained IRB exemption approval. We also acknowledge that our use of the ArtEmis dataset is under ArtEmis Terms of Use 5