{'fraction_non_alphanumeric': 0.03778319725727212, 'fraction_numerical': 0.015268794401442052, 'mean_word_length': 4.6633306645316255, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Modeling and Forecasting Intraday Market Returns: a Machine Learning Approach

In this paper we examine the relation between market returns and volatility measures through machine learning methods in a high-frequency environment. We implement a minute-by-minute rolling window intraday estimation method using two nonlinear models: Long-Short-Term Memory (LSTM) neural networks and Random Forests (RF). Our estimations show that the CBOE Volatility Index (VIX) is the strongest candidate predictor for intraday market returns in our analysis, specially when implemented through the LSTM model. This model also improves significantly the performance of the lagged market return as predictive variable. Finally, intraday RF estimation outputs indicate that there is no performance improvement with this method, and it may even worsen the results in some cases.


Introduction
The recent advances in high-frequency data estimation are associated not only to the technological development and growing processing capacity of big data, but also to the interest in understanding and predicting the behavior of variables in shorter time spaces. Machine Learning methods have been developed in parallel as increasingly accurate tools for estimating and predicting high-dimensional data. Both these fields of study can be easily accommodated into the economic and financial data environment. The adaptation of these methods to incorporate and update information over time allowed the development of robust predictive methods, progressively more relevant in time series analysis. This paper examines the intrinsic relation between market returns and volatility measures, besides lagged returns themselves. Using minute-by-minute intraday data, we find that the CBOE Volatility Index (VIX) can be a strong predictor for the S&P 500 ETF (SPY) in high-frequency, especially through machine learning models. For each day in the sample, we implement a 30 minute rolling-window estimation procedure to forecast the subsequent minute, totaling 340 estimated market returns in a day. This framework is based on the estimation scheme of Chinco et al. (2019) for a cross-section of stock returns. We use different approaches to estimate these returns, from ordinary least squares (OLS) regression benchmarks to more sophisticated methods, focusing on nonlinear models. To build minute-by-minute machine learning models, we relied on the work of Masini et al. (2021).
Particularly, we apply Long-Short-Term Memory (LSTM) neural networks Hochreiter and Schmidhuber (1997) and Random Forest (RF) models Breiman (2001) to estimate multiple intraday rolling windows, using a range of predictive variables in different settings. LSTM models are a variant of Recurrent Neural Networks (RNNs), that differ from the standard neural networks on the ability to remember the previous states in time. This method is broadly used in applications related to weather data, and speech and writing recognition.
For economic and financial data in particular, we want this time dependence to be present, what makes the LSTM such an attractive model, specially due to its ability to remember what matters and forget what is irrelevant to the model. For a nice application of LSTMs in asset pricing see Chen et al. (2019).
Introduced by Breiman (2001), the Random Forest is an ensemble method, which means it combines several simpler models, producing an optimal improved aggregate version in the end. In this case, the base models are classification or regression trees, a nonparametric method based on the mechanism of recursive partitioning of the space of covariates. The average of the estimates generated by each tree are used to build the final forecast, known as Random Forest. RF models have been shown to be a very competitive forecasting tool. See, for example, Medeiros et al. (2021).
The choice to analyze volatility measures as potential predictors is an important issue in the economic literature. Among others, Corsi (2009) and Patton and Sheppard (2015) established a whole literature on realized volatility (RV ), defined as the daily sum of the cumulative squared returns during business hours of a trading day. The forecasting methodology in this field of research uses predominantly autoregressive structures endowed with long memory to predict realized volatilities. McAleer and Medeiros (2008) provide a extensive review of theoretical developments and empirical applications concerning realized volatility.
Time-series properties of the VIX index are addressed by works like Fernandes et al. (2014), as well as its positive contemporaneous link with the volume of the S&P 500 index. Martin (2017) and Martin and Wagner (2019) explore the predictability of market and stock returns defining an associated volatility index named SVIX that provides a bound on the equity premium perceived by the investor. By associating implied variance (VIX 2 ) and realized variances measures, Bollerslev et al. (2009) introduce the variance risk premium (VRP) as the difference between those two variables, that can be used to explain the aggregate stock market returns. Bekaert and Hoerova (2014) extend these results by decomposing VIX 2 into a conditional variance of the stock market and the equity variance premium. The authors conclude that the variance premium is a significant predictor of stock returns.
This plethora of applications for volatility measures and machine learning methods raises our interest in combining these fields of study in a high frequency environment to predict future market returns. As will be discussed throughout the paper, our outputs indicate that machine learning models as the LSTM may improve the performance of benchmark linear regression models in a minute-by-minute framework. Furthermore, if we choose the right regressors, particularly the VIX in our study, the predictive ability becomes even higher.
On the other hand, Random Forests did not bring improvements to our estimations, when compared to the established benchmarks.
The paper is organized as follows. Section 2 reviews the methodology to be implemented in the machine learning framework. The high-frequency data details are presented in Section 3.
Section 4 discusses the employment of different predictive models, as well as the benchmarks to our exercise. The results of intraday estimations and aggregate outputs are described in Section 5. Finally, we conclude the paper in Section 6.

Methodology
Define the following high-frequency forecasting model, where t is the day, and m the intraday minute: where X t,m :" pY t,m´1 , . . . , Y t,m´p , Z 1 t,m , . . . , Z 1 t,m´r q 1 is a n-dimensional vector of predictors, with p ě 1 and r ě 0. F h : R n Ñ R is an unknown measurable function and U t,m`h :" Y t,m`h´Fh pX t,m q has zero mean and finite variance.
For any chosen model framework and forecasting horizon h " 1, . . . , H, we want to define the target function F h , to be estimated from the available data set. This function can be an ensemble of multiple models, and it can vary according to the horizon h. Masini et al. (2021) expatiate in detail about the different machine learning methods, as well as their advances and applications in time series data environment. The authors discuss both linear and nonlinear models, of which we will focus on the last. The reason for this is that sometimes the linearity hypothesis may not encompass all the characteristics of variables such as volatility measures, specially in a high-frequency environment. In this case, we look for alternatives in the universe of models in statistical learning literature.

Long-Short-Term Memory Neural Networks
Recurrent Neural Networks (RNNs) are neural networks that allow for feedback among the hidden layers. RNNs can use their internal state (memory) to process sequences of inputs.
A generic RNN can be written as where p Y t,m`h|m is the prediction of Y t,m`h given observations only up to minute m at day t, f and g are functions to be defined and H t,m is what we call the (hidden) state. From a time-series perspective, RNNs can be see as a kind of nonlinear state-space model.
RNNs can remember the order that the inputs appear through its hidden state (memory) and they can also model sequences of data so that each sample can be assumed to be dependent on previous ones. However, RNNs are hard to be estimated as they suffer from the vanishing/exploding gradient problem. Fortunately, there is a solution to the problem proposed by Hochreiter and Schmidhuber (1997): the Long-Short-Term Memory (LSTM) network . Figure 1 shows the architecture of a typical LSTM layer. A LSTM network can be composed of several layers. In the figure, red circles indicate logistic activation functions, while blue circles represent hyperbolic tangent activation. The symbols "X" and "+" represent, respectively, the element-wise multiplication and sum operations. The RNN layer is composed of several blocks: the cell state and the forget, input, and ouput gates. The cell state introduces a bit of memory to the LSTM so it can "remember" the past. LSTM learns Algorithm 1 describes how the LSTM cell works. f t,m represents the output of the forget gate. It is a combination of the previous hidden-state (H t,m´1 ) with the new information (X t,m ). Note that f t,m P r0, 1s and it attenuates the signal coming com c t,m´1 . The input and output gates have similar structure. Their goal is to filter the "relevant" information from the previous minute as well as from the new input. p t,m scales the combination of inputs and previous information. This signal will be then combined with the output of the input gate (i t,m ). The new hidden state will be an attenuation of the signal coming from the output gate. The prediction is a linear combination of hidden states. Figure 2 illustrates Algorithm 1. Mathematically, RNNs can be defined by the following algorithm: 1. Initiate with c t,0 " 0 and H t,0 " 0.
2. Given the input X t,m , for m P t1, . . . , M u, do: and b c are parameters to be estimated.

Regression Trees and Random Forests
A regression tree is a nonparametric model that approximates an unknown nonlinear function with local predictions using recursive partitioning of the space of the explanatory variables (predictors).
The idea of regression trees is to approximate F h pX t,m q in (2.1) by From the above expression, it becomes clear that the approximation of F h p¨q is equivalent to a linear regression on J dummy variables, where I j pX t,m q is a product of indicator functions.
Let J and N be, respectively, the number of terminal nodes (regions, leaves) and parent nodes. Different regions are denoted as R 1 , . . . , R J . The root node at position 0. The parent node at position j has two split (child) nodes at positions 2j`1 and 2j`2. Each parent node has a threshold (split) variable associated, X s j t , where s j P S " t1, 2, . . . , pu. Define J and T as the sets of parent and terminal nodes, respectively. Figure 3 gives an example. In the example, the parent nodes are J " t0, 2, 5u and the terminal nodes are T " t1, 6, 11, 12u.
Therefore, we can write the approximating model as if the path to leaf i does not include parent node j; 0 if the path to leaf i include the right-hand child of parent node j; 1 if the path to leaf i include the left-hand child of parent node j.
J i : indexes of parent nodes included in the path to leaf i. θ i " tc k u such that k P J i , i P T and ř jPJ B Ji pX t,m ; θ j q " 1.
Random Forest (RF) is a collection of regression trees, each specified in a bootstrap sample of the original data. The method was originally proposed by Breiman (2001)

Data
Our dataset consists of 1 minute frequency data for both the S&P 500 ETF (SPY) and the CBOE Volatility Index (VIX), gathering a total of 3,000 business days between January 2005 and December 2016. We filter the data within each day to the observations between 09:40 and 15:50 (inclusive).
We calculate log-returns of SPY for each minute of day t, using a rolling five-minute return scheme, as follows: where r t,m 2 :m 1 represents the log difference in prices computed between minutes m 1 and m 2 , and P t,m is the price of SPY at day t and minute m. We explore the VIX index variable in level, and implement a transformation of the annualized VIX to an intraday minute variable, in order to facilitate the interpretation of our results 1 . Using this data set, we also compute the one-minute squared version VIX 2 m and first difference ∆VIX m .
Inspired by Bollerslev et al. (2009), we generate a measure of what could be a highfrequency analogous to the variance risk premium, as the difference between the squared one-minute SPY return and the VIX 2 at minute m: The following estimations will be centered on models that use volatility measures as inputs, particularly the VIX. Despite presenting additional configurations and benchmark models, we focus on understanding how the minute-by-minute VIX relates to very highfrequency market returns, as well as its overall predictive capability across the models.
To illustrate the common behavior of market return and the volatility measure over the 252q: 252 business days in a year and 1440 minutes in a day. years, Figure A.1 displays the contemporary relation between the average minute-by-minute SPY log-return and VIX for each day of the sample. The darker the dots in the figure, the more recent the analyzed sample. It is noticeable that the relationship between both variables becomes less disperse over time, more concentrated in recent years, exhibiting lower returns for lower levels of volatility. Additionally, we present the summary statistics for minuteby-minute SPY log-returns and the annualized VIX in Table B.1, which shows that both variables are skewed right and leptokurtic. As expected, market returns and the VIX are negatively correlated.
Despite the clear intrinsic relationship between market returns and near-term volatility implied by stock index option prices, we want to investigate the forecasting potential of SPY returns in high-frequency using volatility measures. The subsequent section presents the fundamental estimation framework adopted in this paper, based on a minute-by-minute rolling window mechanism. This approach is similar to the one developed by Chinco et al. (2019), and allows the researcher to understand very short-term effects over intraday periodicity variables.

Empirical Analysis
To estimate our models, we implement a 30-minute rolling window estimation scheme for very high-frequency predictions. For each minute m and day t, we use the previous nonoverlapping thirty minute window to forecast the one-minute-ahead five-minute log-return of SPY. At the end of each day t, we run a total of 340 estimations between 09:40 and 15:50 2 .
It is important to point out that our high frequency analysis uses non-overlapping input and output variables. This means that for each minute-by-minute log return r t,m 2 :m 1 , we use lagged regressors that do not overlap minutes m 2 to m 1 . For example, if we want to run a regression of the five-minute log return (r t,m´4:m ) on the lagged ∆VIX, we compute the non-overlapping variable as ∆VIX t,m´6:m´5 " VIX t,m´5´V IX t,m´6 . This strategy prevents from using future information to forecast minute m`h when observations are available only up to minute m.
Machine learning algorithms perform better when numerical input variables are scaled.
In each window, we perform a standard MinMax scaling to the data, by using the parameters from the train observations to scale both train and test data 3 . This transformation rescales variables into the range r0, 1s. We implement the standardization to the entire estimation data set, explored in both machine learning and benchmark models, as described in sequence.

Machine Learning Models
The architecture of the LSTM network allows the model to learn and forecast long sequences of data, a particularly useful attribute for time-series analysis. This approach can be put into action by running rolling window regressions in a one-shot multi-step framework for each day t. There are many options available to the researcher in terms of network structure and hyper-parameters to be chosen, including the number of layers, hidden states, loss function and optimizer. In our paper, we optimize the hyper-parameter by tuning it according the models performance on random subsamples.
As discussed in Section 2.1, Random Forest models (which we will also refer as RF from now on) fit classifying decision trees on data subsamples and employ an averaging approach to enhance the predictive power of the data. This ensemble learning method bootstraps the observations in blocks, using randomly sampled training sets (estimation intraday windows, in our case), controlling for potential overfit of the model. The choice of the number of trees to be used is a critical decision in a Random Forest, once it represents the main tuning engine in the model framework.
For each machine learning model (LSTM and RF), we establish three groups of predictors X t,m . The first one is composed exclusively by the lagged volatility variable X t,m " 3 For each window ω rescale:x train ω,m " x train ω,m´m inpx train ω q maxpx train ω q´minpx train ω q andx test ω,m " x test ω,m´m inpx train ω q maxpx train ω q´minpx train ω q V IX t,m´5 , while the second contains only the lagged five-minute return X t,m " r t,m´9:m´5 .
We call these models VIX and AR(1), respectively. Finally, the last group gathers all the variables described in Section 3: X t,m "`r t,m´9:m´5 , r 2 t,m´9:m´5 , V IX t,m´5 , ∆V IX t,m´6:m´5 , V RP t,m´5˘1 , which we labeled as the aggregate model.

Benchmark Models
As benchmarks, we chose different setups for X t,m , implementing the same rolling-window approach. In addition to applying a traditional autoregressive model of order 1, we use lagged squared return, VIX, ∆VIX, and V RP as predictors. We estimate the coefficients in each window through standard ordinary least squares (OLS) regressions.
We define the benchmark models as follows: OLS-AR(1): r t,m´4:m " α m`βm¨rt,m´9:m´5`εt,m OLS-RV: r t,m´4:m " α m`βm¨r 2 t,m´9:m´5`ε t,m OLS-VIX: r t,m´4:m " α m`βm¨V IX t,m´5`εt,m OLS-∆VIX: r t,m´4:m " α m`βm¨∆ V IX t,m´6:m´5`εt,m OLS-VRP: r t,m´4:m " α m`βm¨V RP t,m´5`εt,m (4.1) We use the logic presented in the beginning of this section in order to regress the market returns on non-overlapping lagged variables. The benchmarks OLS-AR(1) and OLS-VIX are particularly important to our analysis, once it can be directly compared to the first two groups of predictors defined in the previous section. From both machine learning and benchmark models, we expect to understand the importance of the lagged log-return, as well as the VIX, in high-frequency forecasting.
This estimation practice was executed in a cloud environment, applying scikit-learn , tensorflow , and keras Python libraries to implement machine learning methods. We employ Python 3.9.4 version to perform the estimations and R 4.1.2 version for output analysis. In the next section, we compare the performance metrics for both machine learning and benchmark models. We contrast specifically the regressions over different groups of predictors presented in Section 4.1 to the single input OLS models in the set of equations 4.1.

Estimation Results
Due to the high dimension of the estimated intraday outputs in each model (described in Section 4), we summarise our results through two performance metrics widely used in the literature: the daily Root-mean-square error (RMSE) and out-of-sample R 2 . To obtain the mean and median RMSE, as well as the standard deviation, we calculate the daily measure pr t,m´rt,m q 2 (5.1) and then compute its overall metric throughout the sample. Then, we calculate the same out-of-sample performance metric implemented by Welch andGoyal (2008) andCampbell andThompson (2008) for each day t R 2 OOS,t " 1´ř M m"1 pr t,m´rt,m q 2 ř M m"1 pr t,m´rt,m q 2 (5.2) to generate the average R 2 OOS , its median, and standard deviation across the days. This metric is relevant to introduce a relation between the mean squared error of our model estimate and the naive projection, represented by the in-sample historical mean within each estimation window. If the error of the model's prediction is lower than the error of the naive forecast, the R 2 OOS is necessarily positive. The same holds for the opposite case, in which a negative R 2 OOS represents a better performance of the historical average model.

Intraday Analysis
To exemplify how each predictive model works throughout a day, Figures A.2 and A.3 illustrate the intraday rolling prediction for a randomly selected date t (January 8, 2007).
Each plot represents a machine learning model (LSTM and Random Forest) compared to the actual series, the historical mean based prediction, and the respective VIX or AR(1) benchmark. At each day t calculate the daily performance metrics for each one of those series, and then compute the average, mean, and standard deviation for the entire sample.
The top plot in Figure A.2 presents the LSTM-VIX model (that uses VIX as predictive variable) minute-by-minute predictions, while the bottom one displays the same scenario for the RF-VIX model. Comparing both to the OLS-VIX benchmark, and the naive projection, one can see that the LSTM method tracks the observed series more smoothly, but also more precisely, while the Random Forest follows pretty much the same pattern established by the benchmark model. For this specific date, the R 2 OOS for the LSTM, Random Forest, and benchmark models were 21.15%,´37.10%, and 6.33%, respectively. We can see a clear superiority of the LSTM model, followed by the benchmark and, finally, the Random Forest model.
This pattern is repeated in Figure A.3, in which we perform the same analysis comparing autoregression based models. The minute-by-minute series forecasts follow an akin behavior to that observed in the previous figure. The curves are smoother, but also more precise for LSTM-AR(1) predictions. Even more interesting, the model now presents a positive R 2 OOS , compared to the negative ones obtained through the RF-AR(1) and OLS-AR(1) models on the same date: 6.93%,´21.87%, and´14.03%, respectively. Once again, the Random Forest output suggests a poor performance for this model. Obviously, these results are restricted to the analysis of a random day in the sample. The subsequent tables present and summarize the general outputs obtained by the estimation of the full sample, leading to similar conclusions.

Overall performance
Tables B.2 and B.3 present the average, median, and standard deviation of R 2 OOS and RMSE, respectively. These outputs are calculated over the performance metrics, using out-of-sample predictions for each day t, as described above 4 . Looking only at the bottom panel of each  distributions present a reasonable performance in general, apart from the RF-AR(1). Still, these models are not able to overcome the OLS-VIX, which stands out as the best performing benchmark.
In a complementary way, Figure A.5 pursues to unravel the relationship between the results obtained by each machine learning model, compared to their respective benchmarks over the years. The vertical axis of each plot represent the LSTM and Random Forest models estimations using as regressor the AR(1) (in the first column of plots) and the VIX (in the second column). The horizontal axis displays the predicted values for the benchmark models. This visualization allows one to identify when machine learning models were able to overcome its benchmarks (the points below the 45°dotted line). The last years in our sample seem to present lower and less disperse estimation errors for all the models combinations.
Moreover, the VIX-based models got even better recently, when using both LSTM and RF models. These highlights synthesize the objective of this paper: contribute to the shortterm forecasting literature, as well as the implementation of modern forecasting techniques through nonlinear rolling window models in a high-frequency setting.

Conclusion
There is growing interest in understanding how to predict high-frequency stock returns. Our paper introduces an innovative approach to estimate minute-by-minute market return (SPY) forecasts using volatility measures and lagged returns as predictors. In addition to standard benchmark models, we implement nonlinear machine learning methods as an attempt to capture idiosyncrasies of this kind of data, whose properties can differ substantially from variables at lower frequencies.
The outputs obtained in our estimations indicate a preliminary, but also encouraging path to better understand how to predict high-frequency market returns. We focus on neural networks (Long-Short-Term Memory) and tree-based (Random Forests) models to estimate multiple intraday rolling windows, using different regressors configurations. Models that use the Cboe Volatility Index (VIX) as predictor stand out, indicating that the VIX is a strong candidate predictor, when compared to other variables. The precision of the forecasts obtained using the OLS benchmark model gets even higher when we apply the LSTM to predict market returns using exclusively the VIX. Although the random forest model is not able to outperform the results obtained by the previous models (eventually even worsening it in some cases), the LSTM proved to be very promising in terms of predictive power, for both VIX and past market returns as regressors.  The plots above represent the intraday rolling prediction of minute-by-minute log-returns on the market. As detailed in Section 4, we run 340 estimations between 09:40 and 15:50, where the first prediction takes place at 10:11 (after the first 30 minute window). The plots represent the LSTM and Random Forest VIX-based models compared to its respective OLS-VIX benchmark, as well as the naive projection. The sample period extends from January 2005 to December 2016. The plots above represent the intraday rolling prediction of minute-by-minute log-returns on the market. As detailed in Section 4, we run 340 estimations between 09:40 and 15:50, where the first prediction takes place at 10:11 (after the first 30 minute window). The plots represent the LSTM and Random Forest AR(1)-based models compared to its respective OLS-AR(1) benchmark, as well as the naive projection. The sample period extends from January 2005 to December 2016. The figure presents the density plots for the daily R 2 OOS distribution generated by each model. The first row of plots display machine learning models, divided by the groups of regressors described in Section 4.1, while the second row exhibits benchmark OLS models detailed in equations 4.1. The sample period extends from January 2005 to December 2016. The plots above represent each machine learning model (vertical axis) against its respective benchmark (horizontal axis). The first column of plots displays the OLS-AR(1) model, while the second reports the OLS-VIX model. The rows of plots represent the LSTM and Random Forest models using AR(1) and VIX as single regressors, respectively. The sample period extends from January 2005 to December 2016.  The table reports mean, median, and standard deviation of the out-of-sample R 2 (R 2 OOS ) throughout the sample. The top panel reports the estimation outputs of machine learning models, while the bottom panel displays the ordinary least squares regression benchmark models, detailed in Section 4. For each metric, the columns represent the groups of predictors described in Section 4.1: VIX, AR(1), and Aggregate. The sample period extends from January 2005 to December 2016. The table reports mean, median, and standard deviation of the Root-mean-square error (RMSE) throughout the sample. The top panel reports the estimation outputs of machine learning models, while the bottom panel displays the ordinary least squares regression benchmark models, detailed in Section 4. For each metric, the columns represent the groups of predictors described in Section 4.1: VIX, AR(1), and Aggregate. The sample period extends from January 2005 to December 2016.