{'fraction_non_alphanumeric': 0.032336100137600424, 'fraction_numerical': 0.013848817080207732, 'mean_word_length': 4.881608145150763, 'pattern_counts': {'":': 0, '<': 18, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Learning Probability Distributions in Macroeconomics and Finance

We propose a deep learning approach to probabilistic forecasting of macroeconomic and financial time series. Being able to learn complex patterns from a data rich environment, our approach is useful for a decision making that depends on uncertainty of large number of economic outcomes. Specifically, it is informative to agents facing asymmetric dependence of their loss on outcomes from possibly non-Gaussian and non-linear variables. We show the usefulness of the proposed approach on the two distinct datasets where a machine learns the pattern from data. First, we construct macroeconomic fan charts that reflect information from high-dimensional data set. Second, we illustrate gains in prediction of stock return distributions which are heavy tailed, asymmetric and suffer from low signal-to-noise ratio.


Introduction
Despite advances in data availability, theory, and computational power, economics have not enjoyed dramatic improvements in forecast accuracy of economic variables over past decades (Stock and Watson, 2017). A fundamental problem underlying the lack of success is that economic variables are difficult to forecast at its nature. Economic forecasters whose decisions depend on such uncertainty hence need to focus on communicating full predictive distribution of the variable surrounding point estimates. 1 At the same time, economists being keen to use large number of series to understand fluctuations in economic data collect data unimaginable decades ago. 2 With the explosive growth in data volume, velocity, and variety, necessity to unlock the information hidden in big data is becoming a key theme in economics (Diebold, 2020). Challenged by proliferation of parameters, heavy critique of arbitrarily chosen restrictions on reduced as well as structured models in past decades, economists wishing to explore possibly rich information content of new datasets recently turned their hopes towards machine learning (Mullainathan and Spiess, 2017). In this paper, we explore use of machine learning for information-rich uncertainty forecasts. We develop a distributional machine learning methods based on deep learning and recurrent network techniques to provide probabilistic forecasts that reflect time series dynamics of possibly large amounts of available information. Such data-driven probabilistic forecasts aim to improve current state in forecasting and communicating uncertainty in economics.
Uncertainty surrounding any step in decision-making is key to understand for financial operations, central and retail banking as well as researchers and practitioners trying to minimize the risk of their decisions, appropriate plans and assisting in the design and implementation of economic policies. Yet even after decades of research, a conditional mean forecast often serves economists as a convenient tool for measuring the central tendency of a target variable, or simply as a best guess about future outcomes of a variable. Variance forecast accompanying it often serves as best expectation about uncertainty and future risk. Such predictions are nevertheless not fully informative in case a decision maker is facing asymmetric dependence of her loss on outcomes from possibly non-Gaussian variables. With an uncertainty being key ingredient in economic decision making, shift to probabilistic forecasting also shifts our hopes towards obtaining better expectations about entire distributions of economic variables. A nontrivial question is how do we make such forecasts, especially utilizing available data.
Traditionally, distribution forecasts are made using time-series models, surveys, or are collected in real-time. 3 With rapid improvements in accessibility and availability of large datasets we believe one can improve description of uncertainty substantially utilizing methods that focus on learning patterns from data. In line with recent endeavors of economists to move away from exclusive dependence on models towards machine learning approaches (Athey and Imbens, 2019) when it makes sense to utilize data and improve our understanding of the problem (Mullainathan and Spiess, 2017), we propose to use deep machine learning to learn the complex patterns in data and return the user a prediction of a full distribution.
A key idea of (machine) learning that can be thought of as inferring plausible models to explain observed data recently attracted number of researchers who document how learning patterns from data can be useful. 4 Surge in the literature and increasing number of applications in economics focus mostly on cross-sectional data and ultimately on point forecasts. While machine can use such models to make predictions about future data, uncertainty plays fundamental part. At the same time, data being key ingredients of all machine-learning systems are useless on their own until one extracts knowledge or inferences from them. Shifting focus from point forecasts towards probabilistic forecasting using big data is essential next step for economists wishing to explore what computer science has to offer.
We contribute to this debate by exploring machine learning in time series context and developing a machine learning strategy to forecast the full distributions in possibly large 3 Methods for constructing distribution forecasts are reviewed in a special issues on "Density Forecasting in Economics and Finance" (Timmermann, 2000) and "Probability Forecasting" (Gneiting, 2008) for collection of papers. dimensional setting. We argue that deep learning in combination with recurrent neural network offers useful tool for distribution prediction without the need of model specification, simply learning the distributions from data. While ability to outperform alternative methods on specific data sets in terms of out-of-sample predictive power is valuable in practice, such performance is rarely explicitly acknowledged as a goal to be addressed in econometrics. As Mullainathan and Spiess (2017) highlights, some substantive problems are naturally cast as prediction problems, and assessing their goodness of fit on a test set may be sufficient for purpose of the analysis. We believe that distribution prediction task in data-rich environment is one of such important problems in economics where machine learning could be helpful for a researcher, policy maker or a practitioner.
What are the challenges specific to probabilistic forecasting of economic variables? Time series as stock returns, electricity prices, traffic data, or macroeconomic series display distributions that can not be captured by convenient Gaussian distribution and hence are not fully characterized by means and variances. These distributions show heavy tails, they are asymmetric, and often violate stationarity. Further, the data contain irregularities, difficult to predict spikes, and regime shifts. Hence complete information about probability of future outcomes given the past information that can be mapped into different representations to construct prediction intervals or probability distribution functions reflecting the data are needed. Such fully approximated distribution function provides a comprehensive information about uncertainty of future observations.
Vast majority of studies that focus on the prediction of conditional return distributions characterize the cumulative conditional distribution by a collection of conditional quantiles (Engle and Manganelli, 2004;Žikeš and Baruník, 2016). In contrast, Leorato and Peracchi (2015) argue that collection of conditional probabilities that describe the cumulative distribution function using set of separate logistic regressions (Foresi and Peracchi, 1995) provide better approach. Following decades resulted in few contributions exploring distributional regressions (Chernozhukov et al., 2013;Fortin et al., 2011;Rothe, 2012) including attempts to overcome problem of monotonicity of the forecasts (Anatolyev and Baruník, 2019) using ordered logistic parametrization. Another important strand of literature focuses on Bayesian forecasting where uncertainty is characterized by probabilities automatically (Geweke and Whiteman, 2006;Lahiri et al., 2010). At the same time, litera-ture in computer science attempts to use machine learning in prediction of distributions.
These attempts are similar to traditional methods and mostly rely on approximation of some pre-specified distribution, such as first two moments. 5 To the best of our knowledge, literature have not yet moved to fully non-parametric approaches approximating the data structures in the context of distributional forecasting in economics and finance.
Why we should believe that machine learning can improve probability forecasts? Classical time series econometrics (Box et al., 2015;Hyndman et al., 2008) mainly focuses on predetermined autocorrelation or seasonality structures in data that are parametrized. With large amount of time series available to researchers, these methods quickly become infeasible and unable to explore more complex data structures. Keeping in mind a famous wisdom that "all models are wrong..., but some of them are useful." (Box et al., 1987), modern machine learning methods are able to easily overcome these problems. Being a powerful tool for approximation of a complex and unknown data structures (Kuan and White, 1994), these methods can be useful in number of application problems where data contain rich information structure and we can not describe it satisfactorily by a simplifying model.
Overcoming the longstanding problem of computational intensity of such data-driven approach with advances in computer sciences adds to the temptation in using these methods for addressing new problems such as distribution predictions.
Our main contribution to the literature is that we propose how to use deep learning techniques as an useful tool for approximation and prediction of conditional distributions in data-rich environment. Our distributional neural network combines deep learning with 5 Duan et al. (2019) applies the natural gradient boosting algorithm to estimate parameters for conditional probability distribution, while assuming homoskedasticity. Salinas et al. (2020) build an autoregressive recurrent neural network, which learns mean and standard deviation for Gaussian, and mean and shape parameter for Negative binomial. Lim and Gorse (2020) classifies price movements for high-frequency trading via deep probabilistic modelling when optimizing parameters of different families of distribution. Although similarly to Salinas et al. (2020),  proposes to use a deep temporal convolutional neural network to estimate parameters of Gaussian distribution to model probabilistic forecast, and they further propose to use the same architecture for non-parametric estimation of quantile regression. The second approach is distribution-free and can produce more robust results. Another study forecasts distributions via direct quantiles using recurrent neural network, Wen et al. (2017) also perform a multi-horizon predictions. Quantile function represented by spline combined with recurrent neural network proposed by Gasthaus et al. (2019) is a distribution-free approach with objective function based on CRPS score (Gneiting and Raftery, 2007) constructed with respect to monotonicity of quantile function. Hu et al. (2019) build deep neural networks to obtain distribution-free probability distribution where one of the steps in the procedure is to obtain cumulative distribution estimates. Januschowski et al. (2020) provide a detailed discussion about ML methods for forecasting. The text discusses way of distinction between "statistical" and "ML" methods adapted in time. recurrent neural networks, it is capable of predicting an entire distribution of a time series and allows to use large amounts of variables. We frame our approach as a multi-output neural network, which returns approximate probability functions of the distribution. The novelty of our approach hence lies in learning of the entire conditional distribution using (deep) recurrent networks from big data. The proposed network is also capable to capture time-variation of distributions when, for example, dealing with highly dynamic data and recover longer and more complex time dependence structures present in data. Our framework generalizes binary choice models (Foresi and Peracchi, 1995;Anatolyev and Baruník, 2019) and together with the state-of-the-art machine learning tools forms a new toolkit for economists interested to describe future uncertainty of economic variables. An important contribution is also our novel approach to construction of objective function that fulfills monotonicity of distributional forecasts by introducing penalty function for divergence from monotone behavior.
Two distinct and important economic datasets illustrate how the machine learning approach to probabilistic forecasting may help a decision maker facing uncertainty. First, we use deep learning to construct data-driven macroeconomic fan charts reflecting information contained in large number of variables. Such data-rich fan charts are first of its kind to reflect high-dimensional information of 216 relevant variables and are of great importance for policy makers as they reflect the structures in data and are not influenced by choice of the model. A forecasting model is in contrast learned from data. Such data-rich fan charts moreover can not be obtained with traditional methods. Second, we study the set of most liquid U.S. stock returns that display asymmetric, heavy-tailed dynamically evolving distributions that are hard to predict due to very low signal to noise ratio.

A Route Towards Probabilistic Forecasting via Deep Learning
Let us consider an economic time series y t collected over t = 1 . . . , T. The main objective is to approximate the conditional cumulative distribution function F(y t+h |I t ) as precisely as possible, and use it for h-step-ahead probabilistic forecast made at time t with information I t containing past values of y t as well as, possibly, past values of other exogenous observable variables.
Consider a partition of the support of y t by p > 1 fixed thresholds corresponding to set of empirical α j -quantiles {q α j } p j=1 where 0 < α 1 < α 2 < . . . < α p < 1 are p regularly spaced probability levels on a unit interval [0, 1]. These partitions are further time-varying, thus in general the elements of the partition are indexed implicitly by t.
The main goal then is to approximate a collection of conditional probabilities corresponding to the empirical quantiles such as F(q α 1 ), . . . , F(q α p ) = Pr y t+h ≤ q α 1 |I t , . . . , Pr y t+h ≤ q α p |I t for the collection of thresholds 1, . . . , p. One convenient way of estimating such quantities is distributional regression. Foresi and Peracchi (1995) noted that several binary regressions serve as a good partial description of the conditional distribution. To estimate conditional distribution, one can simply consider distribution regression model where Λ : z → [0, 1] is a known (monotonically increasing) link function, such as logit, probit, linear, log-log functions 6 and β(.) is an unknown function-valued parameter to be determined. In contrast to estimating separate models for separate thresholds, Cher-  and Baruník (2019) propose to tie the coefficients of predictors in an ordered logit model via smooth dependence on corresponding probability levels. While being able to forecast entire distribution and keeping 0 < F j < 1 and 0 < F 1 (.) < F 2 (.) < . . . F p (.) < 1, the approach still depends on heavy parametrization suited for a specific problem of the time series considered making it an infeasible approach for larger number of variables.

(Deep) Machine Learning
Such probabilistic forecasts heavily depend on the model parametrization and with growing number of covariates become quickly infeasible. Stationarity of data at hand is also requirement that complicates forecasts as it is hard to achieve in many cases. In sharp contrast to such approach, we propose more flexible and general way to the distribution regression via deep learning. We propose a novel multiple output neural network we refer to as a distribution neural network (DistrNN). Our approach aims to uncover nonlinear and mostly complex relationship of time series without specifying strict parametric structure and without requiring strict assumptions about data, while focusing on the outof-sample predictive power of the model.
Machine learning has a long history in economics and finance (Hutchinson, Lo, and Poggio, 1994;Kuan and White, 1994;Racine, 2001;Baillie and Kapetanios, 2007). At its core, one may perceive machine learning as a general statistical analysis that economists can use to capture complex relationships that are hidden when using simple linear methods. As emphasized by Breiman et al. (2001), maximizing prediction accuracy in the face of an unknown model differentiates machine learning from the more traditional statistical objective of estimating a model assuming a data generating process. Building on this, machine learning seeks to choose the most preferable model from an unknown pool of models using innovative optimization techniques. As opposed to traditional measures of fit, machine learning focuses on the out-of-sample forecasting performance and understanding the bias-variance trade-off; as well as using data driven techniques that concentrate on finding structures in large datasets. Further, if one dismisses the "black-box" view of machine learning as a misconception (Lopez de Prado, 2019), it seems nothing should stop a researcher from exploring the power of these methods to solve problems like probabilistic forecasting. However the problems in economics differ from a typical machine learning applications in many aspects. In order to enjoy the benefits of machine learning, a user needs to understand key challenges brought by data. 7 Deep feedforward networks, also often called feedforward neural networks, or multilayer perceptrons lie at heart of deep learning models and are universal approximators that can learn any functional relationship between input and output variables with sufficient data Kuan and White (1994). As a class of supervised learning methods, these approaches are used for classification, recognition and prediction. While being increasingly popular in economics for solution of particular problems Athey and Imbens (2019); Mullainathan and Spiess (2017), probabilistic forecasting have not been explored by the literature yet.
This motivates us to reformulate distribution regression into a more general and flexible distributional neural network. The functional form of the new network is driven by data and we may relax assumptions on the distribution of the data, parametric model as well stationarity of data. The proposed distributional neural network is, as feed-forward network, a hierarchical chain of layers that represents high-dimensional and/or non-linear input variables with the aim to predict the target output variable. Importantly, we approximate the conditional distribution function with multiple outputs of the network as set of probabilities jointly.
As a first step, we exchange a known link function from Eq. 1 for an unknown general function g that will be approximated by a neural network: Next, we consider a set of probabilities corresponding to 0 < α 1 < α 2 < . . . < α p < 1 being p regularly spaced levels that characterize conditional distribution function using set of predictors z t = (y t , x 1 t , ..., x n t ) , and model them jointly as where g W,b is a multiple output neural network with L hidden layers that we name as distributional neural network: It is important to note that in sharp contrast to the literature, we consider a multiple output (deep) neural network to characterize collection of probabilities. Before discussing the details of estimation that allow us to keep monotonicity of probabilities, we illustrate the framework. Figure 1 illustrates how l ∈ 1, ..., L hidden layers transform input data into a chain using collection of non-linear activation functions g (1) , . . . , g (L) . A commonly used activation functions, g ( ) or g (u) = tanh(u). In case g W,b (z t ) is non-linear, neural network complexity grows with increasing number of neurons m, and with increasing number of hidden layers L and we build a deep neural network. We use activation function g (L) (·) = σ(·) to transform outputs to probabilities. Note that for L = 1, neural network becomes a simple logistic regression.
Hidden layer Output layer F(y t+h |z t )

Figure 1. Distributional (Deep) Feed-forward Network.
An illustration of a multiple output (deep) neural network g W,b (z t ) to model the collection of conditional probabilities Pr y t+h ≤ q α 1 |z t , . . . , Pr y t+h ≤ q α p |z t with set of predictor variables z t = (y t , x 1

(Deep) Recurrent Neural Networks
Predictors used by economists often evolve over time, and hence traditional neural networks assuming independence of data may not approximate relationships sufficiently well. Instead, a Recurrent Neural Network (RNN) that takes into account time series behavior may help in the prediction task. Taking into account sequential nature of data that evolve over time and possess an autocorrelation structure, RNNs are more suitable for many economic problems. In contrast to plain neural networks, hidden layers in recurrent networks are being updated in a recurrence for every time step of the sequence meaning that the weights of the network are shared over the sequential data, and hidden states remember the time structure.
Formally, RNNs transform a sequence of input variables to another output sequence with lagged (memory) hidden states
Hidden layer Output layer F(y t+h |z t )

Figure 2. Distributional (Deep) Recurrent Network.
An illustration of a deep recurrent neural network g W,b (z t ) that captures relationship between all nodes (solid) and recurrent paths (dashed) in the network at time t to model the collection of conditional probabilities Pr y t+h ≤ q α 1 |z t , . . . , Pr y t+h ≤ q α p |z t with set of predictor variables z t = (y t , x 1 t , ..., x n t ) . With large number of hidden layers L the network is deep. temporal dynamics that a vanilla RNN is not capable of. Additional depth in capturing the complexity of a tie series can be added by stacking LSTM on top of each other.
Formally, at each time step a new memory cell c t is created taking current input z t and previous hidden state h t−1 and it is then combined with forget gate that controls an amount of information kept in the hidden state as The term σ(·) • c t−1 introduces the long-range dependence, k t is new information flow to the current cell. The forget gate and input gate states control weights of past memory and new information. In the Figure 2, c t is the memory pass through multiple hidden states in the recurrent network.

Loss Function
Since we aim to estimate the cumulative distribution function that is a non-decreasing function bounded on [0, 1], we need to design an objective function that minimizes differences between targets and estimated distribution as well as imposes non-decreasing property of the output. Since the problem is essentially a more complex classification problem closely related to logistic regression, we use a binary cross-entropy loss function. Moreover, to order the predicted probabilities, we introduce a penalty to the multiple output classification problem.
The loss function is then composed of two parts: traditional binary cross-entropy and a penalty imposing monotonicity of predicted output: where (u) + is a rectified linear units function, ReLU, (u) + = max{u, 0}, which passes through only positive differences between two neighboring values, j and j + 1, of CDF, those violating the monotonicity condition, and I {.} is an indicator function. This violation is controlled by the penalty parameter λ m . Note that in addition to its simplicity, ReLU is used for convenience reasons allowing for general use. 8

Networks Design and Estimation Steps
Due to the high dimensionality and non-linearity of the problem, estimation of a deep neural network is a complex task. Estimation requires optimal selection of parameters to provide good performance and avoid potential risks such as over-fitting or convergence problems. Moreover, each problem and data require specific careful choices to minimize the risks. Here, we provide a detailed summary of the model architectures and their estimations.

Learning, Regularization and Hyper-parameters
Selection of hyper-parameters together with regularization methods play crucial role in reduction of risk from estimation. In particular, we use ReLU activation function to introduce non-linearity to our problem, and help the optimization algorithm converge faster.
For the learning process, we use adaptive gradient algorithm, Adam (Kingma and Ba, 2014) and its modification AdamW (Loshchilov and Hutter, 2019) that allows for regularization by decoupling the weight decay from the gradient-based update. The regularization is close to L 2 -regularization with improved results. 9 We use hyper-optimization algorithms based on random search over a grid/cube of parameter ranges, which are specific to a given experiment. For some experiments one might search over the full grid of parameters testing all possible combinations, which can be costly. Using hyper-optimization 10 we select the learning rate of the optimizer, η, the weight decay parameter of AdamW, λ W , and a Dropout parameter regularizing models (Srivastava et al., 2014) that is an efficient way of performing model averaging with neural networks. Specifically, Dropout parameter turns-off a fraction φ ∈ (0.0, 1.0) of nodes in the layer of network at which it is applied. In an in-sample training part, the model is given number of epochs to learn on the data. The number of epochs depend on the size of the data and the batch size used in the estimation. However, we also use early stopping technique that helps regularization to prevent from over-fitting. The early-stopping, or patience, criterion is the minimum number of epochs provided to model to learn.

Code Implementation
We have estimated our models on 48 core Intel® Xeon® /i7 Gold 6126 CPU@ 2.60GHz, 128 GB of memory, and GeForce 3090 GPU. We implement the models using Flux.jl (Innes et al., 2018) package in JULIA 1.6.0. language.

Data Preparation and Information set
To predict the distribution function of a time series y t with observations y 1 , . . . , y T , we This is the maximal time-span providing historical information, in our case, up to the last observation of the validation subsample. The importance here lays in finding the empirical quantiles, q α , corresponding to the set of probabilities {α 1 , . . . , α p }, which are used to build the sequence of target values. Given the information about {y t } t 0 t=1 and the empirical quantiles, we are able to model the distribution conditional on the information set up to time t 0 . Given the information set, we face the problem with non-variation or updating the conditional empirical quantiles for the future distributions. Although, we do not assume a shape of the distribution, we assume, to some extent, small level of shift in the distribution. Further, choice of empirical quantiles and probability levels faces the same problem as quantile regression when it comes to small samples of data.

Empirical Application: Macroeconomic Fan Charts in Era of Big Data
Macroeconomic fan charts are popular tool for communicating uncertainty surrounding economic forecasts. Recognizing the need of communicating uncertainty to public, the Bank of England started to publish fan charts in 1996 and quicky became a leader in communicating uncertainty. Yet the art and science of such important tool for policy making remains on shoulders of the methods chosen.
Here we aim to construct a data-driven macroeconomic fan charts from a best approximating model learned from hundreds of variables with deep learning. This is in sharp contrast to the literature providing uncertainty of macroeconomic variables using so-called prediction fan charts (Britton et al., 1998;Stock and Watson, 2017) using few variables with parametrized and structured model that requires number of assumptions.

Data
To construct such tool for measurement of uncertainty, we use a high-dimensional dataset of McCracken and Ng (2020) that has been extensively used in the macroeconomic literature (Coulombe et al., 2020) and is available at the Federal Reserve of St-Louis's web site. From several alternatives, we opt for the harder to predict quarterly data FRED-QD.
Our dataset contains 216 quarterly US macroeconomic and financial indicators observed from 1961Q1 until 2019Q4. Since number of variables are non-stationary, we follow the transformation codes used by McCracken and Ng (2020). Using this dataset we construct a data-rich fan charts for real GDP growth (GDPC1), inflation (CPIAUCSL), unemployment rate (UNRATE) that will to the best of our knowledge be the first of its kind to reflect highdimensional information of 216 relevant variables. To contrast the data-driven fan charts, we use an state-of-the-art macroeconomic model based on Bayesian Vector Autoregression that contains data-driven factors to incorporate the big data information (McCracken and Ng, 2020).

Deep-learning Based Fan Charts
In order to obtain an h-step ahead forecasts that will form a fan chart, we consider direct prediction scheme. Exploring the data structures, we form h distributional networks where entire (continuous) h-step-ahead conditional distribution g (h) W,b (z t ) is obtained by interpolation of cumulative distribution function preserving the monotonicity of the outcome. Here we apply Fritsch-Carlson monotonic cubic interpolation (Fritsch and Carlson, 1980), for details see Appendix A and use the predicted cumulative distribution function F t+h (α|I t ) to form k − size prediction intervals for a fan chart as such that 1 − k = α u − α l is size of the interval.
To show how useful our approach is, we contrast the predictions obtained from dis-tributional network with the Bayesian vector auto-regression estimated on the factors extracted from data as in McCracken and Ng (2020). This benchmark is a state-of-the-art approach in macroeconomics and at the same time uses information from the whole dataset, hence the forecasts are comparable. To obtain fan chart (prediction intervals), we use the best performing recursive (iterative) scheme y t+h = f (y t+h−1 |I t ) where the prediction intervals are based on the distribution of residuals. Formally, when we assume normal distribution we obtain h-step ahead α prediction interval as where φ(1 − α/2) is corresponding quantile of the std. normal distribution, and, for example, for a naive forecast we haveσ h =σ √ h andσ h is the residual standard deviation. 11 We evaluate the h-step-ahead forecasts with quantile loss function (Clements et al., 2008) for a model m, horizon h, and α-quantile where e t+h,m = y t+h − F −1 t+h,m (α|I t ) is the difference between the original time series and α-quantile forecast given the information set, I t . To compare the predictive accuracy of the models, we use the Diebold-Mariano test (Diebold and Mariano, 1995) with Newey-West variance for h > 1 cases and test the null hypothesis H 0 : L α,m 1 > L α,m 2 against the alternative that m 2 is less accurate than m 1 .

Setup
Working with quarterly data, we compute h = 1, . . . , 6 horizon forecasts each quarter of the out-of-sample period starting at 2012:Q3 and ending with 2019:Q4. Conditional distribution is approximated with j = 1, . . . , 19 empirical α j = (0.01, ..., 0.99) probability levels. The learning explores 36 combinations of hyper-parameters to find the best approximating model for each h-step ahead forecast separately. The hyper-parameters space is optimized once on the training part of the data, and the training procedure performs growing-window forward-validation scheme on the training data using 3-folds. We split the training data while training each fold of validation on train and test parts by ratio 0.93.
We present predictions for deep recurrent neural networks with two hidden LSTM layers of different numbers of neurons chosen in the hyper-optimization. Table A1 in Appendix summarizes all parameters and details used in the estimation. To compare the deep-learning based fan charts, we perform the standard estimation procedure for the Bayesian Vector Autoregression (BVAR) model with factor components as in McCracken and Ng (2020). We use the information criteria 12 and choose the model with four lags. Further, we find the prediction intervals for GDP growth (GDP), inflation (CPI), and the unemployment rate (UNE). The data for both procedures are transformed according to McCracken and Ng (2020) codes and standardized to normal with zero mean and standard deviation one.

Discussion
We start the discussion by presenting the qualitative results of gdp growth, inflation and unemployment predictions in form of fan charts. Figure 3 compares median as well as 50% and 90% prediction intervals made at four different periods by both recurrent distributional neural network and bayesian vector auto-regression approaches and highlights the benefits of deep learning-based predictions.
The prediction intervals from distributional neural network are asymmetric, and in contrast to traditional time series represented by BVAR are not very smooth over forecast horizon. With growing uncertainty when looking ahead into future the deep learning still learns some structure from data and probability intervals resemble less the form of "fans". The intervals are narrower in comparison to BVAR in most of the cases, especially when looking at 50% intervals. In case of unemployment rate, the BVAR model is less capable to reduce the uncertainty about the future observation most probably because of strong spikes in previous years. In contrast, our DistrNN with LSTM units captures the uncertainty precisely.
While Figure 3 is illustrative, it only shows few periods and to support the gains of deep learning approach we further quantify the prediction differences for whole out-ofsample period. Tables 1 and 2 present the quantitative comparison of predictions from both models. We compare the forecasts at h = 1, . . . , 6 horizons using tick loss (Eq. 9) for selected α = {0.1, 0.25, 0.5, 0.75, 0.9} probability levels.
In Table 1  While deep-learning provides us with better forecasts in most of the cases, number of these cases is also significantly different from a traditional BVAR approach. Table 2 reports the results supporting the relative performance of the two methods. Deep learning delivers significantly better prediction in comparison to BVAR in 12 cases while BVAR never outperforms the deep-learning approach significantly with exception of 5-step-ahead forecast of inflation at 25% quantile level. We note that the results depend on 25 out-ofsample observations, which is a size that might be limiting for the test. Stars indicate statistical significance that ***, **, * correspond to 1%, 5%, 10% levels, accordingly. Negative sign of the DM statistics states that DistrNN has better OOS performance than BVAR, positive sign states opposite. We report the out-of-sample forecasts for 25 quarters for three variables GDP growth (GDP), Inflation, and Unemployment rate, at α-levels {0.1, 0.25, 0.5, 0.75, 0.9}, horizons h = 1, . . . , 6, starting at Q3/2013 and ending at Q4/2019.

Empirical Application: Conditional Distributions of Asset Returns
Stock returns data are notoriously known to contain heavy tails and low signal-to-noise ratio (Fama, 1965;Israel et al., 2020). Despite the large literature uncovering these empirical properties, only few studies attempt to forecast the distribution of returns. 13 Among the few, Anatolyev and Baruník (2019) parametrize a simple ordered logit to deliver the distribution forecasts.
Here we aim to build a machine-learning based alternative that is capable of exploring large number of informative variables. We compare the forecasts to the benchmark Anatolyev and Baruník (2019) (henceforth AB) model to see how well the machine learning approach approximates the parametrized model, but we mainly focus on using more variables that classical models can not explore due to its in-feasibility connected with large parameter space.
This application hence serves as a good complement to the previous one where we used machine learning for multiple variable forecast using big data. In contrast, liquid stock returns are known to be hardly predictable, and hence even small improvement is valuable.

Data and Estimation
Our dataset includes 29 most liquid U.S. stocks 14 of S&P500. The main reason for this particular choice is comparability of the results with Anatolyev and Baruník (2019). The daily data covers the period from July 1, 2005 to August 31, 2018. We preprocessed the data to eliminate possible problems with liquidity or biases caused by weekend or bank holidays. The final sample period contains 3261 observations.
We start building the models using the same predictors as in Anatolyev and Baruník (2019) to make a direct comparison of the model forecasts. Specifically, they use Ind t = I {r t ≤q α j } and LogVol t = ln(1 + |r t |) as a proxy to a volatility measure. We will refer to this first choice as the AB predictors. Next, we prepare five realized measures from one-minute intra-day high frequency data obtained from TickData. 15 The realized measures for each of 29 asset returns are realized volatility, skewness, kurtosis, and positive and negative semi-variances labelled as RVol t , RSkew t , RKurt t , RSemiPos t , and RSemiNeg t . These are informative about returns distribution and should help the forecast. We will refer to this set of as RM predictor AB predictors RM predictors AB+RM predictors Table 3. Sets of predictors used in the three models result in over-parametrized model that is not feasible. This is an important note, since our approach provides flexible and more general way of predicting distributions in datarich environment, and at the same time explores possible non-linearity in data. Table 3 summarizes the predictors used in the three models.
Prior to the estimation, we normalize the input data to an appropriate range which eases the job for the algorithm to find better optimum. This is a standard procedure in learning process since the optimization operates on closer ranges while learning in the network structure. Further, we split the data into train and test parts with ratio 0.9 and 0.  Table A2 in appendix B also provides number of epochs allowed to train. This is number of epochs the model is patient about algorithm and waiting on improvement. Finally, we take the model with best validation loss and use it for prediction of out-of-sample distributions. We also study an effect of complexity specifying the size of neural networks. We set the number of nodes from a shallow to deeper DistrNN to [128], [128,64], and [128,64,32]. The network's final layer outputs size is p = 10, which values correspond to probability forecasts approximating the conditional distribution of excess returns given the information set I t .

Statistical Evaluation Measures
We evaluate our probabilistic forecasts using several measures. First, we evaluate the precision of forecasts using the mean square prediction error calculated as where the out-of-sample predicted outputs g W,b,j (z t ) is a matrix keeping dimension of time [t 0 + 1, . . . , T] and conditional probability levels {1, . . . , p}.
To evaluate the compatibility of a cumulative distribution functions with an individual time series observations we use the continuous ranked probability score (CRPS, Matheson and Winkler (1976), Hersbach (2000)): where the conditional CDF g W,b (z t ) is obtained by CDF interpolation (see Appendix A) while the integral is computed numerically using the Gauss-Chebyshev quadrature formulas (Judd (1998), section 7.2) with 300 Chebychev quadrature nodes on [2y min , 2y max ].
CRPS score is of the highest value when the distributions are equal. We obtain an average CRPS score of the out-of-sample forecast as CRPS OOS = 1 T ∑ T t=t 0 +1 CRPS t .
Another measure for the distributional forecast accuracy is Brier score (Gneiting and Raftery, 2007). At time t, it calculates a squared difference of binary realization and the probability forecast, We also compute the average value of the Brier score for the out-of-sample period.
We compare proposed models using relative predictive performance of two models, M 1 /M 2 , where M i is particular measure (MPSE, CRPS, or Brier) corresponding model i.
The model M 1 performs better when the ratio is lower than one.

Discussion
Table 4, Figure 4 and Figure A1 in Appendix B provide all out-of-sample results with the horizon h = 1 comparing sizes of various machine learning models and different variables used as predictors. The performance for all measures is put relative to the maximum likelihood ordered logit model, an AB benchmark of Anatolyev and Baruník (2019).
Overall, we document that machine learning is capable of forecasting conditional distribution of asset returns well, and providing informative variables it delivers improving predictions. Particularly, we observe an average 1% out-of-sample improvements in MSPE for all studied assets in comparison to the parametric models. The improvement is even larger in terms of the Continuous rank probability score evaluating the compatibility of predicted and data distributions.
It is important to note here that in financial forecasting the relationship between statis- tical and economic gains from predictions is non-trivial. Campbell and Thompson (2008); Rapach et al. (2010) note that seemingly small statistical improvement could generate large benefits in practice, which has recently been confirmed on expected returns forecast by machine learning (Gu et al., 2020;Babiak and Baruník, 2020). Hence an average 1 % improvement in the out-of-sample predictions we document will most likely be interesting for a practitioners forming their portfolios based on our forecasts. While it is tempting to explore such strategy, it is far beyond the scope and space of this text.
More specifically, (2020) who find that shallow network performs better than deeper structures one on asset returns data.
While the Table 4 provides aggregate results for all stocks, Figures A1 and 4 complement it with all measures reported for individual assets in boxplots. The detailed look uncovers that machine learning improves performance of individual stocks such as AAPL, AMZZN, GE, or WMT even more. At the same time, Figure A1 shows that in most cases, deeper networks shows lower variance for most of stocks.

Conclusion
In this paper, we have proposed a new approach to modelling probability distributions of economic variables using the state-of-the-art machine learning methods. The distributional neural network relaxes the assumption on the distribution family of time series and lets the model to explore the data fully. The approach is particularly beneficial for modelling data with non-Gaussian, non-linear and asymmetric structures. We show the usefulness of the approach in an economic and financial application. At the same time teh approach is general and can be applied to any other dataset.
We have illustrated that our distributional neural network is useful in constructing big data-driven macroeconomic fan charts that are first of its kind since they are learned from the structure between 216 relevant economic variables. Further, we illustrate how deeplearning can be used to improve probabilistic forecasts of data notoriously know to contain low signal-to-noise ratio, heavy tails and asymmetries.

Appendix for
"Learning Probability Distributions of Economic Variables"

A CDF interpolation
The Fritsch-Carlson monotonic cubic interpolation (Fritsch and Carlson, 1980) provides a monotonically increasing CDF with range [0, 1] when applied to CDF estimates on a finite grid.
Suppose we have CDF F(y) defined at points (y k , F(y k )) for k = 1, . . . , K, where F(y 0 ) = 0 and F(y K ) = 1. We presume that y k < y k+1 and F(y k ) < F(y k+1 ) for all k = 0, . . . , K − 1, which is warranted by continuity of returns and construction of the estimated distribution.

Fixed parameters
The three statistical measures: MSPE (top), CRPS (middle), and Brier (bottom). Each box-plot depicts benchmark values of 29 assets of given NN model size. Anatolyev and Baruník (2019) ordered logit model is benchmark=1. Value lower than 1 states that the purposed model is better that benchmark.