Attention Augmented Convolutional Transformer for Tabular Time-series
  Time-series classification is one of the most frequently performed tasks in
industrial data science, and one of the most widely used data representation in
the industrial setting is tabular representation. In this work, we propose a
novel scalable architecture for learning representations from tabular
time-series data and subsequently performing downstream tasks such as
time-series classification. The representation learning framework is
end-to-end, akin to bidirectional encoder representations from transformers
(BERT) in language modeling, however, we introduce novel masking technique
suitable for pretraining of time-series data. Additionally, we also use
one-dimensional convolutions augmented with transformers and explore their
effectiveness, since the time-series datasets lend themselves naturally for
one-dimensional convolutions. We also propose a novel timestamp embedding
technique, which helps in handling both periodic cycles at different time
granularity levels, and aperiodic trends present in the time-series data. Our
proposed model is end-to-end and can handle both categorical and continuous
valued inputs, and does not require any quantization or encoding of continuous
features.
