Sort Story: Sorting Jumbled Images and Captions into Stories
  Temporal common sense has applications in AI tasks such as QA, multi-document summarization, and human-AI communication. We propose the task of sequencing -- given a jumbled set of aligned image-caption pairs that belong to a story, the task is to sort them such that the output sequence forms a coherent story. We present multiple approaches, via unary (position) and pairwise (order) predictions, and their ensemble-based combinations, achieving strong results on this task. We use both text-based and image-based features, which depict complementary improvements. Using qualitative examples, we demonstrate that our models have learnt interesting aspects of temporal common sense. 
Harsh Agrawal, Arjun Chandrasekaran, Dhruv Batra, Devi Parikh, Mohit   Bansal
cs.CL cs.AI cs.CV cs.LG