Multimodal Categorization of Crisis Events in Social Media
  Recent developments in image classification and natural language processing,
coupled with the rapid growth in social media usage, have enabled fundamental
advances in detecting breaking events around the world in real-time. Emergency
response is one such area that stands to gain from these advances. By
processing billions of texts and images a minute, events can be automatically
detected to enable emergency response workers to better assess rapidly evolving
situations and deploy resources accordingly. To date, most event detection
techniques in this area have focused on image-only or text-only approaches,
limiting detection performance and impacting the quality of information
delivered to crisis response teams. In this paper, we present a new multimodal
fusion method that leverages both images and texts as input. In particular, we
introduce a cross-attention module that can filter uninformative and misleading
components from weak modalities on a sample by sample basis. In addition, we
employ a multimodal graph-based approach to stochastically transition between
embeddings of different multimodal pairs during training to better regularize
the learning process as well as dealing with limited training data by
constructing new matched pairs from different samples. We show that our method
outperforms the unimodal approaches and strong multimodal baselines by a large
margin on three crisis-related tasks.
