A Transformer-based Framework for Multivariate Time Series
  Representation Learning
  In this work we propose for the first time a transformer-based framework for
unsupervised representation learning of multivariate time series. Pre-trained
models can be potentially used for downstream tasks such as regression and
classification, forecasting and missing value imputation. By evaluating our
models on several benchmark datasets for multivariate time series regression
and classification, we show that not only does our modeling approach represent
the most successful method employing unsupervised learning of multivariate time
series presented to date, but also that it exceeds the current state-of-the-art
performance of supervised methods; it does so even when the number of training
samples is very limited, while offering computational efficiency. Finally, we
demonstrate that unsupervised pre-training of our transformer models offers a
substantial performance benefit over fully supervised learning, even without
leveraging additional unlabeled data, i.e., by reusing the same data samples
through the unsupervised objective.
