ChunkFormer: Learning Long Time Series with Multi-stage Chunked
  Transformer
  The analysis of long sequence data remains challenging in many real-world
applications. We propose a novel architecture, ChunkFormer, that improves the
existing Transformer framework to handle the challenges while dealing with long
time series. Original Transformer-based models adopt an attention mechanism to
discover global information along a sequence to leverage the contextual data.
Long sequential data traps local information such as seasonality and
fluctuations in short data sequences. In addition, the original Transformer
consumes more resources by carrying the entire attention matrix during the
training course. To overcome these challenges, ChunkFormer splits the long
sequences into smaller sequence chunks for the attention calculation,
progressively applying different chunk sizes in each stage. In this way, the
proposed model gradually learns both local and global information without
changing the total length of the input sequences. We have extensively tested
the effectiveness of this new architecture on different business domains and
have proved the advantage of such a model over the existing Transformer-based
models.
