{'fraction_non_alphanumeric': 0.024444444444444446, 'fraction_numerical': 0.0044444444444444444, 'mean_word_length': 5.581871345029239, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Improving Classification of Crisis-Related Social Media Content via Text Augmentation and Image Analysis

Identifying, classifying, and prioritizing crisis-related content in social media is an increasingly important task, as users of online platforms continue to expect emergency-response officials to monitor these channels. Much of the current work in this area, however, focuses on applications of neural language models (NLMs) to the text of these social media messages, leaving many meta-content and multi-modal signals unaddressed. This work challenges this textand NLM-centric view as it applies to crisis informatics and the Incident Streams track at the annual Text Retrieval Conference (TREC-IS) by first measuring the performance enhancements NLMs provide over classical text-classification pipelines and then by integrating external data sources and non-textual image analysis. Results suggest classical machine learning (ML) models are still competitive to NLMs, especially in identifying high-priority content, but adding a simple text augmentation strategy results in significant gains in NLM performance. Preliminary results are consistent with the community’s focus on NLMs as our work suggests augmented NLMs perform best in classification, while integrating image analysis has marginal effects on performance. Augmenting samples with inferences from CrisisMMD, however, also significantly increases prioritization performance, suggesting strategies for integrating other data and signals remain valuable. To justify these claims, this paper presents a systematic comparison of classical ML and NLM models for classifying and prioritizing social media content. This paper spans TREC-IS 2020-A and 2020-B, and as such, we are able to validate these various approaches against a held-out set of numerous crisis events and against a collection of COVID-19 events. In this context, we compare our classical ML system from prior TREC-IS editions to several NLM-based systems. We also present a system that integrates image analysis into automated prioritization, for which we currently lack official TREC-IS performance metrics. In particular, this paper describe our research around TREC-IS and answer the following research questions: