Enhancing the Locality and Breaking the Memory Bottleneck of Transformer
  on Time Series Forecasting
  Time series forecasting is an important problem across many domains,
including predictions of solar plant energy output, electricity consumption,
and traffic jam situation. In this paper, we propose to tackle such forecasting
problem with Transformer [1]. Although impressed by its performance in our
preliminary study, we found its two major weaknesses: (1) locality-agnostics:
the point-wise dot-product self-attention in canonical Transformer architecture
is insensitive to local context, which can make the model prone to anomalies in
time series; (2) memory bottleneck: space complexity of canonical Transformer
grows quadratically with sequence length $L$, making directly modeling long
time series infeasible. In order to solve these two issues, we first propose
convolutional self-attention by producing queries and keys with causal
convolution so that local context can be better incorporated into attention
mechanism. Then, we propose LogSparse Transformer with only $O(L(\log L)^{2})$
memory cost, improving forecasting accuracy for time series with fine
granularity and strong long-term dependencies under constrained memory budget.
Our experiments on both synthetic data and real-world datasets show that it
compares favorably to the state-of-the-art.
