{'fraction_non_alphanumeric': 0.02592527932960894, 'fraction_numerical': 0.010867667597765363, 'mean_word_length': 4.8466445521816786, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Deep Kernel Gaussian Process Based Financial Market Predictions

The Gaussian Process with a deep kernel is an extension of the classic GP regression model and this extended model usually constructs a new kernel function by deploying deep learning techniques like long short-term memory networks. A Gaussian Process with the kernel learned by LSTM, abbreviated as GP-LSTM, has the advantage of capturing the complex dependency of financial sequential data, while retaining the ability of probabilistic inference. However, the deep kernel Gaussian Process has not been applied to forecast the conditional returns and volatility in financial market to the best of our knowledge. In this paper, grid search algorithm, used for performing hyper-parameter optimization, is integrated with GP-LSTM to predict both the conditional mean and volatility of stock returns, which are then combined together to calculate the conditional Sharpe Ratio for constructing a long-short portfolio. The experiments are performed on a dataset covering all constituents of Shenzhen Stock Exchange Component Index. Based on empirical results, we find that the GP-LSTM model can provide more accurate forecasts in stock returns and volatility, which are jointly evaluated by the performance of constructed portfolios. Further sub-period analysis of the experiment results indicates that the superiority of GP-LSTM model over the benchmark models stems from better performance in highly volatile periods.


Introduction
It is widely accepted that stock price movement can be predicted to some extent and time series analysis has been a key instrument of portfolio construction and asset allocation.
Empirical research conducted by Marquering and Verbeek (2004) has confirmed the joint economic value of predicting stock market returns as well as volatility. Autoregressive conditional heteroskedasticity (ARCH) model proposed by Engle(1982), Generalized Autoregressive Conditional Heteroskedasticity (GARCH) model proposed by Bollerslev(1986) and relevant extended models allow us to simultaneously describe the process of the conditional mean and volatility for stock returns. However, financial time series data has a significant time-varying characteristic (Giles et al., 2001) which makes the forecasting task a challenging problem.
In recent years, machine learning methods have been widely applied to image identification and natural language processing tasks for looser model assumption and better generalization ability compared with traditional statistical models. More and more researchers are in favor of deploying machine learning methods to solve financial prediction problems.
Compared with the neural networks and the support vector machine, Gauss process (GP) has the advantage of providing probabilistic output. However, the classic GP model has a fixed-form function which limits the fitting ability of this model, so we are motivated to utilize a GP-LSTM model (Al-Shedivat et al., 2017), which extends the basic kernel function of GP with long-short term memory (LSTM) networks.
In this study, we combine the GP-LSTM model with grid search (GS) algorithm to forecast the conditional returns and volatility in financial market. The corresponding trading strategy is designed to prove the effectiveness of the method. The contributions of this paper mainly include the following points: (1) GP-LSTM model has been applied to various fields because this model utilizes the strong fitting ability of LSTM networks while retaining statistical inference ability. To the best of our knowledge, GP-LSTM has not been applied to simultaneously forecast the conditional returns and volatility in financial market.
(2) By combining GP-LSTM model with GS algorithm, this paper minimizes the impact caused by the manual tuning and improve the model performance.
(3) This paper applies this GP-LSTM method to the empirical study covering all constituents of Shenzhen Stock Exchange Component Index (SZSE COMP) and designs the corresponding trading strategy. In addition, Sharpe Ratio of the constructed portfolios is introduced to jointly measure the forecast performance of conditional return and volatility.
The rest of this paper is organized as follows. In second section, we review the related work concentrating on forecasting stock returns and volatility. The third section provides a detailed instruction of the GP-LSTM method, the benchmark models and the process of data preparation. The corresponding experiment results are presented in the fourth section. In the fifth section, further sub-period analysis is conducted to explore the source of GP-LSTM's superiority. The sixth section concludes this paper and points out the possible direction of future research.

ARCH and GARCH family models
ARCH and GARCH family models are classic methods for forecasting financial returns and volatility. To describe the volatility clustering effect of financial asset price, ARCH model assumes that the conditional variance depends on the lagged squared error terms in the volatility equation. However, ARCH family models often need to set a high lag order, which greatly increases the number of parameters required to estimate. GARCH model is correspondingly proposed to solve this problem. As set in the GARCH model, variance of conditional returns depends not only on the lagged squared error terms, but also on lagged terms of the variance itself. In this way, the GARCH models with low lag order can be used instead of the ARCH models with high lag order.
In order to reflect the asymmetry impact of error terms in the volatility equation of GARCH and Nelson (1991), Glosten and Jagannathan (1992) extended the standard GARCH model to EARCH model and GJR-GARCH model respectively. In GARCH family models, Gaussian distribution is generally used to characterize the error term in mean equation, but the distribution of stock returns usually exhibits the characteristic of high peak and fat tail in realistic situations. Therefore, researchers made a natural extension for the error term from Gaussian to non-Gaussian (e.g. t distribution, t skewed distribution) for fitting financial data better.

Artificial Neural Networks
Artificial Neural Networks (ANN) consists of a set of units called artificial neurons, which are usually organized in several layers. According to Universal Approximation Theorem (Hornik et al., 1989), feedforward neural networks can approximate a Borel measurable function to any desired degree of accuracy if sufficiently many hidden units with arbitrary squashing functions are provided. Donaldson (1997) constructed a nonlinear GARCH model based on ANN to predict stock index volatility. Hyup (2007) utilized GARCH model to generate input variables of an ANN to enhance the prediction ability in forecasting volatility. These studies empirically prove that ANN has practical value in the prediction of volatility.

Support Vector Regression
The support vector machine (SVM) proposed by Vapnik (1999) is also one of the most important machine learning models. In classification tasks, SVM classifies the data points by finding a hyperplane that has the maximum margin, i.e. the maximum distance between support vectors. As the linear separability may not be satisfied, the kernel function is introduced to map the data points in raw representation into a higher even infinite dimensional space. The support vector regression (SVR) is an extension of SVM model and transforms the discrete output of SVM into an ordinary 1-dimensional numerical space for regression tasks. Pérez-Cruz et al. (2003) utilized SVM algorithm to estimate the parameters of GARCH model, improving the forecasting accuracy of volatility. Peng (2018) also chose to estimate the mean and volatility equations of GARCH, Egarch and GJR-GARCH models respectively using support vector regression for the data of three cryptocurrencies and three currency exchange rates, which achieved more accurate predictions than the basic GARCH family models.
However, the studies for volatility prediction utilizing ANN or SVR usually choose a proxy variable as an approximation for the real volatility (variance of the conditional return).
Hence, classic machine learning methods can not be directly applied in the prediction of real volatility.

Brief introduction of Gaussian Process
Similar to SVR model, kernel trick is also widely used in GP models. The covariance between each pair of data points, is measured in the kernel space and Bayesian theory is applied to predict the posterior distribution of out-of-sample data points. Compared with ANN and SVR models, the GP model has the advantages of providing probabilistic output and non-parametric inference flexibility.  constructed the Gaussian Process Volatility model based on the standard Gaussian Process and found that it outperformed the classic GARCH models.
However, in the previous applications of SVR and GP model, researchers often choose fixed-form kernel functions, which limits the fitting ability of these models. Al-Shedivat et al.
(2017) proposed a new GP model in which the kernel can be learned by neural networks like LSTM networks and this new model has been successfully applied to many problems such as wind power prediction and automatic driving system. In realistic situations, the stocks always differ in size, industry and liquidity, leading to different modes of stock price movement. Therefore, it's natural to assume that hyper parameters which decide the signal-to-noise ratio and volatile degree should vary widely when applying GP model to different stocks. To solve this problem, GS algorithm can be used to find a good set of initial hyper parameters and reduce the impact of manual tuning.
In this paper, we combine the deep kernel GP model with grid search algorithm for the prediction problem of stock conditional mean and volatility, and construct the portfolio from constituent stocks of the SZSE COMP according to Sharpe Ratio.

Model construction
A GP model could be viewed as a generation process of the Gaussian distribution, and arbitrary finite variables yielded by a Gaussian Process have a joint Gaussian distribution which is determined by a mean function x is a feature vector and The covariance between each pair of data points is determined by the feature vectors j i x x , and the kernel function k . Given the training set ) , ( X Y and out-of-sample points * X , we can predict conditional return and variance of * Y based on Bayesian method by the following formulas: As shown in Fig. 1, we denote the feature vector and output variable at time t as t x , t y respectively in this paper.

Learning the deep kernel
The GP-LSTM model deployed in this paper utilizes a single-layer LSTM with 8 neurons to generate an advanced feature vector containing temporal information as the input of basic kernel function (see Eq. (3)).

Fig. 2 gradient optimization
We choose the Radial Basis Function ( RBF ) as the basic kernel function here, the form of ij K can be specified as: As depicted in Fig.2, the hyperparameters  of the GP layer and the parameters W of the LSTM network will be trained jointly.

Grid Search
In the application of GP-LSTM model, squared-exponential covariance function is chosen as the kernel for the GP layer, which has three hyper parameters: length scale L, the signal variance 2 f  and the noise variance 2 n  . Rasmussen et al. (2005) has illustrated that a Gaussian Process with different hyper-parameter combinations may all allow corresponding model residuals seem like white noise. It means that finding a good set of hyperparameters can be difficult for the RBF kernel. As the hyper-parameter space is infinite, we utilize the heuristic grid search method to choose the optimal combination of initial hyper parameters according to the performance on validation set. Although the hyperparameters are optimized during the training process, choosing proper initial values can still improve the prediction performance. By assembling the above modules together, we utilize the GP-LSTM method ( Fig. 3(a), Fig. 3(b)) to capture the price movement patterns of different stocks and reduce the manual tuning effect.

Training process
As described above, GP-LSTM unit (see Fig.3 (a)) can be divided into two parts: the classical GP regression model and the LSTM network which maps the raw representation of sequential data into advanced feature vector for the kernel function. For the LSTM part, we adopt fixed hyper parameters including learning rate, maximum learning epochs, batch size etc. For the GP part, we go through all the listed hyper-parameter combinations (Hyper parameters 1 -Hyper parameters N) to select the optimal one according to the Mean Squared Error ( Mse ) on the validation set.
Besides the selection of hyper-parameter combination, the remaining parameters of the kernel function K and the LSTM networks need to be trained for fitting and prediction.
Despite the strong ability to fit highly complex function, deep neural networks may easily meet with over-fitting problem. We can utilize early-stopping method to solve this problem.
As shown in Fig.4, the early-stopping requires to evaluate model performance on validation set after each training cycle and the training process will be stopped when the generalization error begins to increase.

Data source
The data source in our study is the market data of constituent stocks in Shenzhen Stock Exchange Component Index (SZSE COMP), which was first released on January 23rd, 1995 and it has been one of the oldest, most complete and influential constituent stock indexes in China's stock market. Our experiment dataset is the daily returns of SZSE COMP constituent stocks from January 1st, 2013 to July 12th, 2017.  We can see from Table 1 that there are significant differences in terms of statistical characteristics like kurtosis between different stocks and the differences indicate the different underlying mechanisms.

Data preprocessing
The raw data used in this paper is the daily stock returns of 38 constituent stocks listed in We specify the feature vector to be 20 lagged terms of logarithmic return and their sign directions while setting the task label to be the stock return one period ahead. In order to facilitate the training process, we firstly normalize the return sequence Hence, the feature vectors and labels of the task can be denoted as   and t i r , respectively.

Generation of training sets and test sets
As mentioned above, the sample period used in this study is from January 1, 2013 to July 12, 2017. We select the last 300 days as the test set, while the remaining data is divided into training set and validation set according to the ratio of 7:3. Table 2 lists the software applied in our experiment. Data preparation is processed by Python 3.6 with the Numpy toolkit and Pandas toolkit. The GP-LSTM unit in our framework is built based on the keras-gp package which extends Keras with GP layers. The GS algorithm is also coded in Python 3.6. Besides, GARCH family models are employed as the benchmarks through the Rugarch package based on R language.

Benchmark Models
Similar to GP model, GARCH family models also can be divided into two parts: the mean equation which depicts the movement of conditional mean, and the volatility equation that depicts the conditional variance of the error term in mean equation. Several commonly used GARCH family models are described in detail as follows.

Standard GARCH model
As mentioned previously, in standard GARCH model, the variance of the error term denoted by t h is impacted not only by the lagged squared error terms

EGARCH model
To reflect the asymmetrical shock, Nelson (1991) proposed the exponential GARCH model (EGARCH) model, which can be written as: where j  reflects the size effect and j  reflects the sign effect.

GJR-GARCH model
The asymmetrical shock of the information flow can also be described by GJR-GARCH model proposed by Glosten, Jaganathan, and Runkle (1993). The GJR-GARCH can mathematically be described as follows: . , 1 ) If leverage effect parameter 0   , the influence of the information shock is asymmetrical.
In this paper, we choose benchmark models to be SGARCH model， EGARCH model and GJR-GARCH model. The error term of each of the three models is set to be Gaussian distribution, t distribution and skewed t distribution respectively. The resulting 9 benchmarks are represented as SGARCH-norm, SGARCH-std, SGARCH-sstd, EGARCH-norm, EGARCH-std, EGARCH-sstd, GJR-GARCH-norm, GJR-GARCH-std, GJR-GARCH-sstd respectively. In addition, we set 1 , 1   q p (see Eq. (10-12)) for these models because this setting is the most widely used in realistic situations.

Results
In this section, we focus on presenting experiment findings and start with the exhibition of the overall performance of the constructed portfolios during 2016/04/20 -2017/07/12. For simplicity, we do not take the transaction fees into consideration and set risk free return to be 0. Then, detailed portfolio performance when 15  k reflects that GP-LSTM method can jointly forecast the conditional returns and volatility accurately. Besides, we compare the forecasting accuracy of different models from other perspectives, such as return prediction Accuracy and test Kupiec VaR . Table 3 in Appendix summarizes the trading strategy performances based on each model.

 k
As shown in Table 4, GP-LSTM model achieves high daily returns on both long position and short position, which results in the highest hedge return among all the models. This indicates that the new framework can effectively capture the characteristic patterns to distinguish the high-return stocks and low-return stocks.
Value at Risk ( VaR ), the estimation of the maximum potential loss under a given confidence level. Darrel et al. (1997) have made a comprehensive review about VaR theory, and point out that this method can effectively measure the financial market risk. From Table 4, we can also find that GP-LSTM model is superior to all benchmarks in the VaR at 5%, 7.5%, 10% confidence level respectively. Fig. 5 plots the accumulative return of each model on the test period with the black and bold one representing the GP-LSTM based strategy.

Mean Squared Error and Accuracy
In order to measure the performance of daily return forecast more directly, we calculate the Mean Squared Error ( Mse ) and Accuracy by the following formulas: A smaller Mse or a higher Accuracy means that we have a more accurate forecast of the daily return. For a certain model, we also define a kind of stocks termed occupying stocks, whose definition is the stocks on which the model can achieve the best performance in a certain metric. We count the dominant stocks of each model in Mse and Accuracy , and make the corresponding ranking. We can see from Table 5

that GP-LSTM method outperforms all other models in average
Accuracy and the number of corresponding occupying stocks. Although the GP-LSTM method is inferior to GARCH family models in Mse , it still ranks high in number of dominant stocks.

Comparison of the VaR forecast results
As the daily volatility is not visible, we usually measure the accuracy of volatility by The greater the P value, the closer the empirical failure rate N n / to the confidence level, which means a higher predictive accuracy of volatility. Table 6 displays the average P value and number of occupying stocks of each model in the test set of 38 stocks for 5%, 7.5% and 10% confidence level respectively.
We can see from Table 6 that GP-LSTM ranks high in most cases except when alpha = 5%. Although EGARCH-sstd is superior to GP-LSTM in terms of VaR test, this model places 6 among the 10 models in metric Accuracy presented in Table 5, and GP-LSTM model places 1 in this metric. Similarly, the GJR-GARCH-norm model ranks first in the overall Mse and the number of occupying stocks in Mse , but in terms of direction Accuracy and VaR prediction, GP-LSTM model is still the superior one. Generally speaking, GP-LSTM method provides an accurate forecast for both conditional return and volatility, which results in the significant performance of the constructed portfolio.

Further discussion
To further analyze the models' performance under different market conditions, this paper splits the backtesting period of 300-day time horizon into two sub periods equally: April 20th, -November 29th, 2017and November 30th, 2016-July 12th, 2017. The comparison and analysis of the 10 models over the two periods are conducted in this part.

Sub-period performance
It can be observed from Fig. 6 that, the GP-LSTM method exceeds other models by a large margin with respect to accumulative return during the early half period (2016/04/20 -2016/11/29). During the period of 2016/11/30 -2017/07/12, GP-LSTM still has a stable performance. Placing 4 among all the models, GP-LSTM model does not produce an evident advantage against other models, which is significantly different from the performance on the early half period. In the nest section, we will give an in-depth analysis about this difference.

Sub-period analysis
For exploring the characteristic of two sub periods mentioned above, we calculate the autocorrelation function ( acf ) and partial correlation function ( pacf ) coefficients for 20 lagged terms. The acf measures the correlation between t y and k t y  while the pacf measures the partial correlation which removes the effects from other lags. Averaging the acfs and pacfs of all the sample stocks on the two sub periods, we plot Fig. 7 and Fig. 8 respectively as follows. By plotting the graphs of averaged acf and pacf coefficients, we can see that the stock data on the first sub period shows a more significant long-term dependency compared with the second sub period. The K-line of the SZSE COMP (see Fig. 9) also illustrates that the former sub period is more volatile than the latter. Therefore, we can conclude that GP-LSTM method has an obvious advantage against all other models in the market condition with higher complexity.

Fig. 9
Daily return of SZSE COMP on the two sub periods

Conclusion
In this paper, we discuss the characteristics of previous models for predicting the conditional return and volatility in financial market. It is found that machine learning methods Further sub-period analysis shows that GP-LSTM method has better generalization ability than the benchmarks in highly volatile and complex periods. Although LSTM networks have been widely used due to the strong fitting ability in modeling sequential data, the input sequence of LSTM is still mapped into a fixed-length vector, which means a loss of information. For future research, we will focus on information compression to promote representation ability for financial time series data and improve the forecasting performance of GP framework.