{'fraction_non_alphanumeric': 0.029681978798586573, 'fraction_numerical': 0.01342756183745583, 'mean_word_length': 4.518662952646239, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Approximating intractable short rate model distribution with neural network

We propose an algorithm which predicts each subsequent time step relative to the previous time step of intractable short rate model (when adjusted for drift and overall distribution of previous percentile result) and show that the method achieves superior outcomes to the unbiased estimate both on the trained dataset and different validation data.


Introduction
Let a stochastic process with the following properties be denoted as S t : where y t = y 0 e −α2t + η t 0 e α2(s−t) dZ 2 (s) (3) where all the parameters including x 0 and y 0 are known in advance. Then S t is normal with the mean E [S t ] = x 0 e −α1t + y 0 e −α2t (4) and the variance Since the distribution is normal, best unbiased estimate is also normally distributed, and should not need moments beyond the first two. The formulation above is a kernel of an existing problem of short rate computation within two factor log-normal models.
r(t) = e xt+yt+ϕ(t) ≡ e St+ϕ(t) Given the distribution of r(t) we can deduce the distribution S t . The problem can be formulated as follows: given the distribution F (S t )(s) = P rob(S t ≤ s) find the distribution F (S t+1 )(s). Under the method of moments F (S t )(s) = F (S t+1 )(s), since this is a theoretical measure. We train a neural network to test, whether on an average sample it can outperform the theoretical F (S t+1 )(s) prediction.
The same Neural network can then outperform the Method of moments in a set generated from different parameters data.
2 Generating training and validation data

Introduction
In order to train the neural network to perform the same functions as stochastic equations, a large data series was generated. In order to improve the precision of computation in line with the possible evolution of paths, we use a branching technique, which increases the number of unique paths by a multiple of 4 at every timestep (discretization of continuous process).

Generating data from Stochastic equations
We generate a large dataset (5,000 simulations based on 4 12 individual realisations of the model, for each simulation). This is achieved by branching each of the outcomes in the previous step by multiple of 4 (i.e. 1st step has 4 results per simulation, the next 16 and so on, until 12th) -see Appendix 2 for more information. In order to derive formulation for our neural network we will have to reduce the model to percentile outcomes.
Given formulation of 2-factor Black-Karasinski model as: where u(0) = 0, dZ 1 dZ 2 = ρ dt and µ , α 1 , α 2 , σ 1 , σ 2 are constants, define a new state variable where: Then we can re-write the original equation as: The above, is a classic formulation of 2 factor Hull-White model. This then allows [3] the G2++ formulation of the model, where the state variables are decoupled. The new formulation will have the following terms: Such that (8) is a deterministic equation relative to time t. Reformulation of the above into two independent Vasicek equations is obvious. The attempt to estimate the original series from the previous data (i.e. decomposition and re-composition independent series) creates additional estimation error with its own noise. Hence in order to reduce the method derived error we will attempt to do the estimation on the joint series.
Explicitly this would mean that each percentile value of the outcome would need to be adjusted by the joint drift over the time step, and then amended for the standard deviation. However, having tested various possible drift functions, we came to conclusion that not introducing any drift to the previous percentile values gives the most stringent version of the test.

Introduction
Typically intractable short rate are approximated by binomial trees [1] which rely on theoretical moments of the function. We use the moments of theoretical distribution to compare the results generated by running the model in the process described in Appendix 2.

Estimating next timestep with Method of Moments
Method of moments in this paper approach describes a theoretical measure: based on the theoretical standard deviation and mean of the distribution we derive the quantiles of the distribution values. The expectations for the next step are easily derived using basic Euler scheme: Standard deviation over a particular time step are derived from orthogonal decomposition of the above model into two independent Vasicek processes. This approach is standard when approximating the distribution of path with binomial trees. This results in the following expression: Given a realization of the process S t from (4) at step t, we can index it by e (e a1 +e a2 )dt in order to arrive at the expected values over the timestep t + 1.
Having adjusted for the mean, we can then compute the realized percentiles from the theoretical standard deviation of the distribution effectively deriving the percentiles from the theoretical first two moments of the distribution. X is an array of F (r t )(s) for s = 0.5%, ..., 100% at t = 2, ..., 12.
Where ... denotes an 2D array shifted by +1 along the t.
Where is defined as the variance and covariance of the two processes 4 Model formulation -Neural Network

Introduction
In order to show the possibility of additional information captured by looking at distribution as a whole we use the simplest neural net possible, and attempt not to overfit the data. We verify the results against the mean scenario, and scale it by standard deviation of resulting from stochastic errors between all scenarios for each percentile. We can create a state space representation of required function of recurrent neural network as such: 1+e −x optimising for D matricies and c vectors with stochastic gradient descend method [2] . Where D 1 is a 10 by 200 matrix, and D 2 is a 200 by a 10 matrix, and c 1 is a 10 parameter vector, and c 2 is a 200 parameter vector.

Neural Network Specification
We will then train a simple (single layer, 10 nodes) neural network to derive additional information (relying on the values of other percentiles) about the distribution from the percentiles of the previous timestep at time t (i.e. n= 200 values corresponding to 0.5% to 100% quantiles of the distribution), to predict the outcomes at time t+1.

Input layer
Hidden layer

Output layer
Such that the trained neural network is able to produce outcomes that lie within 1 standard deviation (this is relative error in graphs below) from true We used MLPRegressors ( [5]). This is a single hidden layer Multilayer Perceptron regressor [4], and limited the number of nodes in a single hidden layer to 10 regressors.
The network uses 10 sigmoid nodes to regress the input (in form of percentile values) against the output (percentile values at t+1). It fits the weights and intercepts to inputs to produce outputs, and this is then repeated for each incremental month from 2nd to 12th, for all 5,000 scenarios.
The notable effect of this is constraining the amount of information carried from one example to the next. It is also useful to note, that when the number of regressors increases the fit is increasingly aligned with the method of moments.
The graphs below present the forecast error on average for each of the quantiles at 3rd, 6th, 9th and 12th timestep standardized by standard error for the 5,000 scenarios, on that dataset. Figure 1 goes here

Simulation
The original hypothesis is that the previous t value of distribution carries no bearing on the way the distribution looks at the next time step t + 1 if this statement is true then the neural network (we train on data at t=0) should not be able to learn any features that would enable it to beat method of moments.
However, here the variance of the process increases with predetermined amount sigma (as per theoretical distribution), so if we have previous distribu-tion correctly standardized against the theoretical distribution in the previous step, the distribution of the next step would be best predicted by the percentile outcomes of the previous step.
As both processes utilize Brownian motion, we can use these percentiles directly to predict the future standardized percentiles. By relying on sigmoid/logistic function, we can use the fact that the normal distribution is relatively well approximated by the logistic equation in terms of CDF and derive the new values of CDF. We evaluate the quality of fit on the mean of the process (which is not part of training dataset), in order to remove stochastic error from the evaluation process. We also standardize the estimation by stochastic error of each percentile. In order to verify the veracity of the method we test it against a different dataset (not used for training, and with 1000 trails on different σ 1 , α 1 , σ 2 , α 2 , r 0 values) to verify that the method does indeed produce superior performance to the method of moments. (In the case where only part of these parameters are changed, the case for using Neural Networks is much stronger). This result holds across different time points across the 12 month evolution of the process.

Conclusion
Compared with method of moments the neural network produces lower RMSE. Due to constant changes in the calibration results this method is superior, because it's more accurate and faster.
Two things of note: the errors appears to have a constant structure which suggests training a time dependent neural net, or indeed a simple regression may further reduce the error. The fit of the method of moments is better around the mean, hence a further process can be added to place more reliance on theoretical mean around that point.   We take advantage of the fact that values the are closer to the point of origin require less dispersion -I.e. there are fewer significantly different paths, but this number grows with time.

Data availability statement
Data available on request from the primary author