TENT: Tensorized Encoder Transformer for Temperature Forecasting
  Reliable weather forecasting is of great importance in science, business, and
society. The best performing data-driven models for weather prediction tasks
rely on recurrent or convolutional neural networks, where some of which
incorporate attention mechanisms. In this work, we introduce a novel model
based on Transformer architecture for weather forecasting. The proposed
Tensorial Encoder Transformer (TENT) model is equipped with tensorial attention
and thus it exploits the spatiotemporal structure of weather data by processing
it in multidimensional tensorial format. We show that compared to the classical
encoder transformer, 3D convolutional neural networks, LSTM, and Convolutional
LSTM, the proposed TENT model can better learn the underlying complex pattern
of the weather data for the studied temperature prediction task. Experiments on
two real-life weather datasets are performed. The datasets consist of
historical measurements from weather stations in the USA, Canada and Europe.
The first dataset contains hourly measurements of weather attributes for 30
cities in the USA and Canada from October 2012 to November 2017. The second
dataset contains daily measurements of weather attributes of 18 cities across
Europe from May 2005 to April 2020. Two attention scores are introduced based
on the obtained tonsorial attention and are visualized in order to shed light
on the decision-making process of our model and provide insight knowledge on
the most important cities for the target cities.
