{'fraction_non_alphanumeric': 0.03347424133072138, 'fraction_numerical': 0.01195999587586349, 'mean_word_length': 4.697669864891326, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 1, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
SSDNet: State Space Decomposition Neural Network for Time Series Forecasting

In this paper, we present SSDNet, a novel deep learning approach for time series forecasting. SSDNet combines the Transformer architecture with state space models to provide probabilistic and interpretable forecasts, including trend and seasonality components and previous time steps important for the prediction. The Transformer architecture is used to learn the temporal patterns and estimate the parameters of the state space model directly and efficiently, without the need for Kalman filters. We comprehensively evaluate the performance of SSDNet on five data sets, showing that SSDNet is an effective method in terms of accuracy and speed, outperforming state-of-the-art deep learning and statistical methods, and able to provide meaningful trend and seasonality components.


I. INTRODUCTION
Time series forecasting is an important task in many practical applications, e.g. predicting electricity demand, stock prices, immune response and disease progression over time.
Statistical methods such as ARIMA and exponential smoothing are well established for time series forecasting and are a part of the more general framework of State Space Models (SSMs) [1]. SSMs are considered interpretable models as components such as trend and seasonality can be extracted and used for decision making and explanation. However, they are not able to infer shared patterns from a set of related time series as each time series is fitted independently [2], [3].
Deep learning methods have been recently investigated as a promising alternative, due to their ability to learn from raw data with minimum domain knowledge and to extract complex patterns, including shared patterns across related time series. Prominent examples include DeepAR [4], a probabilistic forecasting model based on Long Short Term Memory (LSTM) neural networks, DeepSSM [5] which combines SSM with LSTM, Deep Factor models [6] which combine LSTM with a local probabilistic model, and LogSparse Transformer [3] and Informer [7], which are modifications of the Transformer architecture [8] for time series forecasting. However, deep learning models are difficult to interpret; generating interpretable forecasts is crucial for adopting these systems in practical applications.
Approaches for producing interpretable forecasts using deep learning models have been recently proposed. For example, Guo et al. [9] and Li et al. [10] explored the structure of LSTM and DeepSSM attempting to learn the variable importance for prediction. Oreshkin et al. [2] developed the deep learning architecture N-BEATS. They showed that its interpretable configuration (N-BEATS-I) is able to successfully learn and output the trend and seasonality components of the forecast, which is useful for forecasting practitioners. However, N-BEATS is designed for univariate time series, it provides point forecasts and has limited ability to model trends with varying slope because it uses a deterministic trend model. In this paper, we present a new forecasting approach to address these issues.
The main contributions of this work are as follows: 1) We present a new forecasting approach, called State Space Decomposition Neural Network (SSDNet), which combines the Transformer deep learning architecture with a SSM. SSDNet combines the advantages of deep learning (learning from raw data without intensive feature engineering and ability to infer shared patterns from related time series) with the interpretability of SSM models. It employs the Transformer architecture to learn the temporal pattern and directly estimate the parameters of SSM. To facilitate interpretability, we used a fixed form SSM to provide trend and seasonality components and the attention mechanism of the Transformer to identify which parts of the past history are most important for the prediction. 2) We evaluate the performance of SSDNet on five time series forecasting tasks. The results show that SSD-Net achieved higher accuracy than the state-of-the-art deep learning models DeepAR, DeepSSM, LogSparse Transformer, Informer and N-BEATS and the statistical models SARIMAX and Prophet. SSDNet was also able to provide interpretable results by showing nonlinear trend and seasonality components. To evaluate the effectiveness of the fixed form SSM part, we conduct an ablation study by replacing the Transformer with LSTM, and this architecture also showed competitive results.

II. PROBLEM FORMULATION
We consider three tasks: 1) solar power forecasting, 2) electricity demand forecasting and 3) exchange rate forecasting. Accurate forecasting of the generated solar power and the electricity demand is needed for optimal scheduling of generators and large-scale integration of solar into the electricity grid. Over-prediction may lead to wasting energy, under-prediction may result in blackouts. Exchange rate forecasting is used to derive future monetary value, earn profits and avoid risks in international business environments.
Sanyo and Hanergy contain solar power generation data from two PV plants in Australia -from 01/01/2011 to 31/12/2016 (6 years) for Hanergy and 01/01/2011 to 31/12/2017 (7 years) for Sanyo. Only the data between 7am and 5pm is considered and was aggregated at half-hourly intervals. For both datasets, weather and weather forecast data was also collected (see [16] for more details) and used as covariates. Solar contains solar power data from 137 PV plants in Alabama, USA, from 01/01/2006 to 31/08/2006. Electricity contains electricity consumption data for 370 households from 01/01/2011 to 07/09/2014. The Solar and Electricity data is aggregated into 1-hour intervals. Exchange contains daily exchange rate data for 8 countries from 01/01/1990 to 31/12/2016. Following [3], [16], [17], calendar features are added according to the granularity of the datasets: Sanyo and Hanergy use month, hour-of-the-day and minute-of-thehour, Solar uses month, hour-of-the-day and age, Electricity uses month, day-of-the-week, hour-of-the-day and age and Exchange uses month, day-of-the-week and age. Fig. 1 shows the partial autocorrelation plots for each data set. Fig. 1 (a) shows the results for both Sanyo and Hanergy since they have a similar partial autocorrelation pattern; both datasets include only 1 time series. Fig. 1 (b)  show the results for Solar, Electricity and Exchange datasets respectively. All plots are for 8 time series for consistency since 8 is the common number of time series in the three datasets, see Table I. Fig. 2 shows the additive decomposition results for a period of 7 days for Sanyo, Hanergy, Solar and Electricity. For Exchange we just show one time series as this dataset doesn't have a clear seasonality and additive decomposition methods cannot be applied.
We can see that all time series except Exchange exhibit seasonality and repetitive patterns with high partial autocorrelation, and sometimes there are also significant short-term trend variations. For the Exchange set, there is no clear repetitive pattern and its partial autocorrelation at lag 1 is the only significant one.
All data was normalized to have zero mean and unit variance.

B. Problem Statement
Given is: 1) a set of N univariate time series (solar, electricity or exchange series) {Y i,1:T l } N i=1 , where Y i,1:T l := [y i,1 , y i,2 , ..., y i,T l ], T l is the input sequence length, and y i,t ∈ is the value of the ith time series (generated PV solar power or consumed electricity) at time t; 2) a set of associated timebased multi-dimensional covariate vectors {X i,1:  where T h denotes the length of the forecasting horizon. The covariates for the Sanyo and Hanergy datasets include: weather and calendar features {Z i,1:T l +T h } N i=1 , while the covariates for the Solar, Electricity and Exchange datasets include only calendar features. Our goal is to predict the future values of the time series {Y i,T l +1:T l +T h } N i=1 , i.e. the PV power or electricity usage for the next T h time steps after T l .
Specifically, SSDNet produces the probability distribution of the future values, given the past history: where Φ denotes the parameters of SSDNet, and the input of SSDNet at step t is the concatenation of y i,t−1 and x i,t . The models are applicable to all time series, so the subscript i will be omitted in the rest of the paper for simplicity.

III. RELATED WORK
DeepSSM [5] combines SSMs with Recurrent Neural Networks (RNNs). A RNN is used to generate the parameters of a linear Gaussian SSM for time series forecasting and Kalman filter is used to derive posterior knowledge. DeepSSM is considered interpretable because it can recover the SSM parameters and they can be inspected. The Deep Factor model DF-LDS [6] is a variation of DeepSSM that can handle non-Gaussian observations and it also uses Kalman filters.
Although Kalman filters have been successfully used in linear dynamic systems for decades, the need for covariance matrix inversions is computationally expensive and limits their applicability for processing large data sets [18], [19]. In contrast, Eleftheriadis et al. [20] proposed Gaussian Process State Space Model (GPSSM) merging Gaussian process with a bidirectional RNN to approximate the posterior of a non-linear system. However, GPSSM has high complexity of inference (O(T ) per time step) which is a disadvantage for forecasting multi-horizon large data sets. N-BEATS [2] is a novel deep neural architecture using backward and forward residual links and stacks of fully connected layers for univariate point forecasting. Each N-BEATS layer contains a number of blocks which produce partial forecasts; these forecasts are aggregated at stack level and then network level in hierarchical fashion. The final forecast is the sum of the partial forecasts of all blocks. N-BEATS can produce interpretable forecasting results without expert knowledge by employing a fixed form of block function to generate prediction with decomposition components. The interpretable version (N-BEATS-I) uses the deterministic polynomial trend model and assumes a stationary trend with constant slope. However, the local trend of time series over a short period (one or a few days) could vary significantly and inconsistently -e.g. the solar power generation or electricity consumption could decrease suddenly due to temporary cloud coverage or irregular activities of users (see Fig. 2). Hence, a stochastic model which assumes a nonstationary trend with varying slope would be more suitable for such cases [21] and this is what we use in SSDNet.
In summary, compared to DeepSSM, the proposed SSDNet: i) employs the Transformer architecture to learn temporal patterns, and estimates the parameters of SSM and the probability term directly instead of using Kalman filters; ii) uses the SSM with fixed and non-trainable transition matrix at the decoder to generate decomposition results within the forecasting horizon, while DeepSSM uses a more flexible SSM along all time steps; iii) combines Mean Absolute Error (MAE) and Negative Log-Likelihood (NLL) together as loss function to achieve accurate point and probabilistic forecasting results simultaneously instead of using NLL only; iv) shows better interpretability by using a stochastic decomposition technique in the form of SSM with an innovation term to model nonlinear trend and seasonality components and attention mappings to show the importance of historical time steps for forecasting. Compared to N-BEATS, SSDNet can model trend with nonlinear slope, provide probabilistic forecasts and also better utilizes the covariates.

IV. SSDNET
The aim of SSDNet is to model temporal patterns effectively in order to provide accurate probabilistic and interpretable forecasts.

A. Network Architecture
Then, we employ additive time series decomposition model in the form of SSM and generate the predictionŷ t at step t by summing up the trend component T t , seasonality component S t and probability component I t . We decompose the times series using SSM and generate the forecasting result as: where the latent state vector α t ∈ s×1 contains trend T r t and seasonality S t , s is the number of seasonality and I t is sampled from the Gaussian distribution with zero mean and σ 2 It variance. Different from the traditional SSMs, we remove the random noise term from the state transition in Eq. (4) to prevent the state noise propagation and estimate the probabilistic term I t directly as shown in Eq. (3). The SSM part of SSDNet does not process historical series directly but instead utilizes the latent component generated by the Transformer and encodes the information of the time steps before the forecasting horizon in the initial state vector α T l . We also introduce the innovation term c t ∈ s×1 in the state transition Eq. (4) to allow SSDNet to learn stochastic trends with fluctuations in time series. This is necessary as the simple summation of linear trend and seasonality components cannot model complex series effectively and would introduce large residuals (see Fig. 2). While the traditional SSMs need to use Kalman filter to update the posterior and derive the covariance matrix to optimize noise terms, SSDNet uses the Transformer architecture resulting in less matrix computation and memory usage.
The innovation term c t and the variance σ 2 It are learnt from the latent factor o t directly, as illustrated in Eq. (6) and (7). The use of the Softplus function ensures that SSDNet always generates positive variance, and the HardSigmoid function [22] is used for a speed-up: The parameters state vector α t , state transition matrix Γ t and result transition matrix z t that determine the structure of SSM in Eq. (3) and (4) are shown below: Note that Γ t and z t are non-trainable and fixed for all time steps. In this work, we assume that the trend follows a random walk process, and other processes such as moving average could be modelled by adjusting the first row of the transition matrix Γ t . The initial trend T r 0 and seasonality S −s+1 , ..., S −1 , S 0 values of the state vector α 0 are unknown and need to be generated based on the latent components, they are learned from historical steps: Alternatively, the forecasting result (3) could be written as the aggregation of the predicted trend, seasonality and variance to provide probabilistic result: where we consider the real-world data follows a Gaussian distribution (other distribution function could be used in Eq. (5) for higher flexibility), and predictions are sampled from the distribution. The ρ-quantile output could be generated via the inverse cumulative probability distribution:ŷ t = F −1 t (ρ). At step t, Eq. (4) could be written in an additive form, where the trend and seasonality evolve by adding the innovation term at each time step, as shown in Eq. (11) and (12). The trend and seasonality models are based on the classic random walk and dummy seasonal, and the change of trend and seasonality are dominated by the innovation variable. The innovation variable for trend can be different at every time step and allow SSDNet to model the nonstationary trend with varying slope. For the extreme case when all innovation terms are zero, the trend would be constant and the seasonality over all periods will be the same.
Theorem 1. ∀t > 0 and t ∈ Z, the range of trend T r t and seasonality S t of SSDNet are bounded: Proof. Assume that innovation term c t , trend and seasonality have maximum values: 1) c t outputs its maximum value of 0.5 at all time steps, 2) the initial trend has its maximum value T r 0 = 0.5 and 3) the seasonality has its maximum value of S 0 = − s−1 j=1 −0.5 = 0.5 × (s − 1). Following Eq. (11) and (12), the trend and seasonality increase using the innovation terms with maximum values of c t [1] = 0.5 and c t [2] = 0.5 respectively at each time step.
Considering the initial values and increments together, the maximum values of the trend and seasonality are (t + 1) × 0.5 and (s − 1 + t) × 0.5 at time step t, correspondingly. Since g c (.) is a symmetric activation function, its lower bound is the negative value of its upper bound, hence the minimum values of trend and seasonality are the negative of their maximum values.
Theorem 1 implies that the upper and lower bound of the SSDNet trend and seasonality components expand in the time domain and SSDNet can model time series where the variation in the seasonal pattern increases over time. The variation of the trend component is significantly smaller than that of the seasonality and helps SSDNet to model a slow-moving trend and significantly varying seasonality. The constraints of both components help SSDNet to learn an inherent structure and allow α t to generate meaningful waveform (observable in our case studies) and enable the interpretability of forecasting results.
In our case since the data is normalized to have zero mean and unit variance, when visualizing trend and seasonality components, the bias (data mean) is added to the trend and both the trend and seasonality are scaled (data variance) back.

B. Loss Function
SSDNet needs to provide both accurate point and probabilistic forecasts. Accordingly, we developed the loss function shown in Eq. (13), which considers both the point and probabilistic forecasts by combining the MAE and NLL using the regularization parameter a: higher a allows the model to pay more attention to the probabilistic forecast. The point forecast isŷ t and it consists of trend and seasonality. The parameters of SSDNet are optimized by minimizing this function.
L(ŷ T l +1:T , σ 2 I T l +1:T , y T l +1:T , a) = a × NLL(ŷ T l +1:T , σ 2 I T l +1:T , y T l +1:T ) + MAE(ŷ T l +1:T , y T l +1:T ) V. EXPERIMENTS We compare the performance of SSDNet with nine other models: six state-of-the-art autoregressive deep learning models (DeepAR, DeepSSM, N-BEATS-G, N-BEATS-I, LogSparse Transformer and Informer), a SSM (SARIMAX), an interpretable regression model (Prophet) and a persistence model: • Persistence is a typical baseline in forecasting and considers the time series of the previous day as the prediction for the next day. For the Exchange dataset we use the last 20 steps of the input sequence. • SARIMAX [1] is an extension of the ARIMA and can handle seasonality with exogenous factors. • Prophet [23] is an interpretable regression model predicting trend, seasonality and holiday components. • DeepAR [4] is a widely used sequence-to-sequence probabilistic forecasting model. • DeepSSM [5] fuses SSM with RNNs to incorporate structural assumptions and learn complex patterns from the time series. It is the state-of-the-art deep forecasting model that employs SSM. • N-BEATS [2] is based on backward and forward residual links and stacks of fully connected layers. N-BEATS-G provides generic forecasting results, while N-BEATS-I provides interpretable results by decomposing the time series into trend and seasonality. We introduced covariates to N-BEATS at the input of each block to facilitate multivariate series forecasting. • LogSparse Transformer [3] is a recently proposed variation of the Transformer architecture for time series forecasting with convolutional attention and sparse attention; it is denoted as "LogTrans" in Table III  iterations. The hyperparameters with a minimum loss on the validation set were selected. The models used for comparison were tuned based on the authors' recommendations. The probabilistic forecasting models use the NLL loss, and the point forecasting model (N-BEATS) uses the mean squared loss.
Following the experimental setup in [3], [16] and [17], we used the following training, validation and test split: for Sanyo and Hanergy -the data from the last year as test set, the second last year as validation set for early stopping and the remaining data (5 years for Sanyo and 4 years for Hanergy) as training set; for Solar and Electricity -the last week data as test set (from 25/08/2006 for Solar and 01/09/2014 for Electricity), the week before as validation set and the remaining data as training set. For the Exchange dataset, we used the 480 working days from 02/02/2015 is test set, the 480 working days before as validation set and the remaining data as training set. For all data sets, the data preceding the validation set is split in the same way into three subsets and the corresponding validation set is used to select the best hyperparameters.
For the Transformer-based models, we used learnable position and ID (for Solar, Electricity and Exchange sets) embedding. For SSDNet, the seasonality s was set to 20 for Sanyo, Hanergy and Exchanges and 24 for Solar and Electricity, the loss function regularization parameter a was fixed as 0.5, the learning rate λ was fixed, the dropout rate δ was chosen from  Table  II and used for the evaluation of the test set.
Following [3]- [5], [16], [17], we report the standard ρ0.5 and ρ0.9-quantile losses. Note that ρ0.5 is equivalent to the Mean Absolute Percentage Error (MAPE) [24]. Given the ground truth y and ρ-quantile of the predicted distributionŷ, the ρ-quantile loss is given by: A. Accuracy Analysis The ρ0.5 and ρ0.9 losses are shown in Table III. As N-BEATS and Persistence do not produce probabilistic forecasts, only the ρ0.5-loss is reported for them. We can see that overall SSDNet is the most accurate model -it outperforms all the other methods on all data sets except Electricity and Exchange.
For the Electricity data set, SSDNet has the best ρ0.9-loss and second-best ρ0.5-loss among the probabilistic forecasting models and is also the best among the interpretable models (N-BEATS-I, DeepSSM and Prophet). For the Exchange data set, SSDNet is the second-best model after SARIMAX.
SSDNet performs better than DeepSSM on all datasets which indicates that the proposed new architecture, loss function and the use of fixed SSM were beneficial.
The two N-BEATS models perform well on Solar and Electricity (univariate data sets) but much worse on Sanyo and Hanergy (multivariate, including the weather features); a possible reason is that N-BEATS was designed for univariate forecasting and our modification to handle multivariate datasets was not sufficiently effective in this case.
We further investigated the performance on the Exchange dataset, for which the deep neural network models, including the proposed SSDNet, performed worse than SARIMAX. This finding is consistent with [25], and the reason is that Exchange is the only data set without repetitive patterns and a seasonal component. Fig. 5 and Fig. 6 show the predictions of SSDNet and SARIMAX for 2 test samples from the Exchange data set. We can see that SSDNet tends to generate predictions with a smooth trend and some variations, while SARIMAX tends to generate predictions that do not vary much with time. Hence, the success of SARIMAX on the Exchange data set can be explained with the characteristics of this data set -it is more stable and does not have strong repetitive patterns and seasonality, as shown in Section IIA and Fig. 1.
Another interesting finding for Exchange is that when we simply use the last time step value Y T l of the input series as the predictions Y T l +1:T l +T h , we obtain ρ0.5-loss of 0.010 which is as good as SARIMAX and outperforms all other models. However, SSDNet and the other deep learning models demonstrate significantly higher accuracy than SARIMAX on the other four data sets which include repetitive patterns.

B. Ablation Analysis
To evaluate the effectiveness of the SSM part of SSDNet, we conducted additional analysis -an ablation study. Table IV shows the performance of SSDNet-LSTM, which is SSDNet with the Transformer part replaced by LSTM. Thus, we keep the SSM part for decomposition and prediction the same but replace the Transformer with LSTM for SSM parameters estimation.
SSDNet-LSTM is compared with DeepAR [4] and DeepSSM [5]. DeepAR uses LSTM to generate predictions directly and DeepSSM uses LSTM and Kalman filter to estimate the parameters of SSM, while SSDNet-LSTM can be considered as a combination of DeepAR and our proposed SSM or as a variation of DeepSSM by replacing the classic SSM with our proposed SSM without a Kalman filter. Table IV shows that SSDNet-LSTM outperforms both DeepAR and DeepSSM, indicating the effectiveness of the SSM part for decomposition and foresting. SSDNet-LSTM is also competitive with SSDNet which shows the robustness and flexibility of our fixed form SSM -its parameters can be

C. Speed Analysis
We evaluate the training and testing time of the two SSDNet models (SSDNet and SSDNet-LSTM) and compare them with DeepSSM which also employs SSM. All models are trained on the same computer configuration; we report the average elapsed time and the standard deviation of 10 runs. Table V and VI show the training time per epoch and testing time in milliseconds for the Sanyo, Hanergy, Solar and Exchange data sets. No speed test was conducted for the Electricity data set as we report the accuracy results from their papers [3], [5] and the values of the hyperparameters used are not known.
Comparing the training speed, we can see that both SSDNet and SSDNet-LSTM, which do not use Kalman filter, are significantly faster than DeepSSM -the average training speed per epoch of DeepSSM is 390.48% and 458.24% times slower than SSDNet and SSDNet-LSTM respectively. In terms of testing speed, DeepSSM is the fastest, very closely followed by SSDNet-LSTM (7.68% slower) and then SSDNet. This is because the Kalman filter in DeerSSM is not required for inference and the architecture of LSTM used by DeepSSM is simpler than the Transformer used by SSDNet.
Overall, the training speed of SSDNet and SSDNet-LSTM is significantly improved due to the changes in SSM and their testing speed is competitive with DeepSSM. Fig. 4 presents the SSDNet output for a 7-day test set sample for the Sanyo, Hanergy, Solar and Electricity data sets, and Fig. 5 presents the SSDNet output for 2 test samples from the Exchange data set. We can see that the results are interpretable: the trend is smooth and monotonically decreasing or increasing, the seasonality shows regular and cyclic fluctuations (except for the Exchange data set which doesn't have seasonality).

D. Interpretability Analysis
SSDNet can model a rapidly changing trend -for example, in Fig. 4 (d), the solar power generation drops rapidly on day 7 and SSDNet correctly outputs a declining trend. SSDNet can not only learn smooth seasonal patterns but also has the ability to learn seasonal patters with small random fluctuations (e.g. at days 1 and 6 of Fig. 4 (a)) due to the introduction of innovation term c t . For the Exchange data set which doesn't have clear seasonality, the peak-to-peak magnitude of the learned SSDNet seasonal component is small. Fig. 7 (a) presents the results for 8 consecutive days from the test set for the Electricity data -7-day past history and 1-day forecasting output of SSDNet, and shows obvious daily  Fig. 7: SSDNet case study on Electricity data set (a) actual vs predicted data; (b) attention patterns of the first head in the last layer and (c) attention patterns of the second head in the last layer seasonality. Fig. 7 (b) and (c) show the attention patterns of two heads in the last multi-head attention layer of SSDNet. The attention mappings show the importance of the previous time steps for predicting the future time steps. The first head tends to attend to the first and seventh day before the predicted day, which is consistent with the weekly cycle of electricity data. The second head attends to almost all days. A possible explanation is that the first head mainly focuses on learning the seasonality, while the second head mainly focuses on learning the trend.

VI. CONCLUSION
We presented SSDNet, a novel deep learning approach, for probabilistic and interpretable forecasting of time series data. SSDNet combines the advantages of deep learning with the interpretability of SSM models. SSDNet employs the Transformer architecture to learn the temporal patterns, extract latent components and estimate the parameters of SSM. It then applies SSM to generate the interpretable forecasting results with nonstationary trend and seasonality components. SSDNet also applies attention mechanism to visualize the important past time steps for the predicted future steps.
We evaluated the performance of SSDNet on five time series forecasting tasks. The results show that in terms of accuracy SSDNet outperformed the state-of-the-art deep learning models DeepSSM, DeepAR, LogSparse Transformer, Informer and N-BEATS, and the statistical models SARIMAX and Prophet. It was also able to provide interpretable results by showing clear trend and seasonality components. Our ablation study showed the effectiveness of the SSM part and that SSDNet with LSTM instead of Transformer also performed well. SSDNet was also much faster to train than DeepSSM. Hence, the results show that SSDNet is a promising method for time series forecasting.