Preformer: Predictive Transformer with Multi-Scale Segment-wise
  Correlations for Long-Term Time Series Forecasting
  Transformer-based methods have shown great potential in long-term time series
forecasting. However, most of these methods adopt the standard point-wise
self-attention mechanism, which not only becomes intractable for long-term
forecasting since its complexity increases quadratically with the length of
time series, but also cannot explicitly capture the predictive dependencies
from contexts since the corresponding key and value are transformed from the
same point. This paper proposes a predictive Transformer-based model called
{\em Preformer}. Preformer introduces a novel efficient {\em Multi-Scale
Segment-Correlation} mechanism that divides time series into segments and
utilizes segment-wise correlation-based attention for encoding time series. A
multi-scale structure is developed to aggregate dependencies at different
temporal scales and facilitate the selection of segment length. Preformer
further designs a predictive paradigm for decoding, where the key and value
come from two successive segments rather than the same segment. In this way, if
a key segment has a high correlation score with the query segment, its
successive segment contributes more to the prediction of the query segment.
Extensive experiments demonstrate that our Preformer outperforms other
Transformer-based methods.
