Time-Stamped Language Model: Teaching Language Models to Understand the
  Flow of Events
  Tracking entities throughout a procedure described in a text is challenging
due to the dynamic nature of the world described in the process. Firstly, we
propose to formulate this task as a question answering problem. This enables us
to use pre-trained transformer-based language models on other QA benchmarks by
adapting those to the procedural text understanding. Secondly, since the
transformer-based language models cannot encode the flow of events by
themselves, we propose a Time-Stamped Language Model~(TSLM model) to encode
event information in LMs architecture by introducing the timestamp encoding.
Our model evaluated on the Propara dataset shows improvements on the published
state-of-the-art results with a $3.1\%$ increase in F1 score. Moreover, our
model yields better results on the location prediction task on the NPN-Cooking
dataset. This result indicates that our approach is effective for procedural
text understanding in general.
