{'fraction_non_alphanumeric': 0.02336007550731477, 'fraction_numerical': 0.008022652194431336, 'mean_word_length': 4.518880208333333, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Noise estimation by use of neighboring distances in Takens space and its applications to stock market data

We present a method that uses distances between nearest neighbors in Takens space to evaluate a level of noise. The method is valid even for high noise levels. The method has been verified by estimation of noise levels in several chaotic systems. We have analyzed the noise level for Dow Jones and DAX indexes and we have found that the noise level ranges from 25 to 80 percent of the signal variance.


I. INTRODUCTION
It is a common case that observed data are contaminated by a noise (for a review of methods of nonlinear time series analysis see [1,2]). The presence of noise can substantially affect invariant system parameters as a dimension, entropy or Lapunov exponents.
In fact Schreiber [3] has shown that even 2% of noise can make a dimension calculation misleading. It follows that the assessment of the noise level can be crucial for estimation of system invariant parameters. Even after performing a noise reduction one is interested to evaluate the noise level in the cleaned data. In the experiment the noise is often regarded as a measurement uncertainty which corresponds to a random variable added to the system temporary state or to the experiment outcome. This kind of noise is usually called the measurement or the additive noise. Another case is the noise influencing the system dynamics, what corresponds to the Langevine equation and can be called the dynamical noise.
The second case is more difficult to analyze because the noise acting at moment t 0 usually changes the trajectory for t > t 0 . It follows that there is no clean trajectory and instead of it an ǫ-shadowed trajectory occurs [4]. For real data a signal (e.g. physical experiment data or economic data) is subjected to the mixture of both kinds of noise (measurement and dynamical).
Schreiber has developed a method of noise level estimation [3] by evaluating the influence of noise on the correlation dimension of investigated system. The Schreiber method is valid for rather small gaussian measurement noise and needs values of the embedding dimension d, the embedding delay τ and the characteristic dimension r spanned by the system dynamics.
Diks [5] investigated properties of correlation integral with the gaussian kernel in the presence of noise. The Diks method makes use of a fitting function for correlation integrals calculated from time series for different thresholds ε. The function depends on system variables K 2 (correlation entropy), D 2 (correlation dimension), σ (standard noise deviation) and a normalizing constant Φ. These four variable are estimated using the least squares fitting. The Diks method [6] is valid for a noise level up to 25% of signal variance and for various measurement noise distributions. The Diks's method needs optimal values of the embedding dimension d, the embedding delay τ and the maximal threshold ε c .
Hsu et al. [7] developed a method of noise reduction and they used this method for noise level estimation. The method explored the local-geometric-projection principle and is useful for various noise distributions but rather small noise levels. To use the method one needs to choose a number of neighboring points to be regarded, an appropriate number of iterations as well as optimal parameters values d and τ .
Oltmans et al. [8] considered influence of noise on the probability density function f n (ε) but they could take into account only a small measurement noise. They used a fit of f n (ε) to the corresponding function which was found for small ε. Their fitting function is similar to the probability density distribution that we receive from correlation integrals 1 N 2 DET n (ε). The method needs as input parameters values of d, τ and ε c .
In Ref. [9] we presented a method of noise level estimation by coarse-grained correlation entropy (NECE). The crucial point of this method is fitting of a proper function to the estimated correlation entropy. This method does not demand any input parameters like the embedding dimension d or the embedding delay τ . The minimal and maximal values of the threshold parameter ǫ can be automatically estimated. The NECE method will be used further as the reference method.
In this paper we present another method for evaluation of a noise level. The method makes use of neighboring distances in the embedding space (NEND) and will be introduced in the next section. In the further section we show an application of this method to stock market data. Although it is a common believe that the stock market behaviour is driven by stochastic processes [10,11,12] it is difficult to separate stochastic and deterministic components of market dynamics. In fact the deterministic fraction follows usually from nonlinear effects and can possess a non-periodic or even chaotic characteristic [13,14]. With the help of the NEND and NECE methods [15] we try to demonstrate that stock market data are not purely stochastic and a deterministic part can be sometimes dominant.

II. METHOD OF NOISE ESTIMATION BY USE OF NEIGHBORING DIS-
sponding d-dimensional vector constructed in the embedded space where d is an embedding dimension and τ is an embedding delay. The method is based on the assumption that the minimal distance between nearest neighbors is described by the standard deviation of noise.
The nearest neighbor is found using the Euclidian norm i.e. the distance is measured using the following formula The nearest neighbor of the vector x n is the vector x j such that We will assume that the distance between the vector x n and its nearest neighbor (dis N N n ) is calculated in a large embedding dimension d >> 1.
For linear systems without a noise the minimal distance between nearest neighbors should decrease with an increasing number of data in time series and for N → ∞ this distance will tend to zero since the trajectory reaches the final periodic orbit. For deterministic chaotic systems such minimal distances depend on the system entropy but they also tend to zero for an infinite number of data when the trajectory densely fills the chaotic attractor.
In the case when we add to an observed deterministic trajectory a Gaussian non-correlated noise the corresponding minimal distance dis N N n is increased. For a large value of the embedding dimension d >> 1 the distances can be estimated as a standard deviation of a superposition of 2d independent stochastic variables, i.e.
where σ is the standard deviation of a noise added to the signal.
The approximation (3) is valid only in limits of very long time series and a large embedding dimension d. If we generate surrogate data {surr i } by the random shuffling of the original data [14] then this kind of surrogates preserves mean, variance and histogram but removes any determinism in data. The minimal distance between nearest neighbours calculated in an embedded space for surrogate data should be proportional to standard deviation of data. Now let us define the Noise-To-Signal ratio as the proportion of σ to the standard devi- In the first step of the method we calculate all distances between nearest neighbors in the original and in the surrogate data. Then we search for the smallest distance for each data set: i.e. the linear dependence of the distance dis N N min on the noise level, we can introduce the output parameter of the method ADET d [16], which is related to the Noise-To-Signal ratio (NT S) as follows: . Returns x n are defined as where P n is the value of an index at the time n. Noise levels for both indexes are in the range NT S ≈ 0.5 − 0.9 as one can see in Figs (2,3,5,6). It follows that considered stock market data are not purely stochastic because the percent of determinism ranges (1 − NT S 2 ) · 100% ≈ 20 − 75% and the stochastic part is about 25 − 80%. In Figs (2,5) we present noise levels ADET 9 calculated with the NEND method for DJIA and DAX indexes We think that the NECE method gives more relevant results in high volatility regions and the NEND method underestimates the noise level in such cases.

IV. CONCLUSIONS
In conclusion we have developed a new method of noise level estimation from time series. The method makes use of the minimal distance between nearest neighbors in Takens space. The method has been tested for several systems and it has brought similar results to the method described in Ref. [9] but it is much easier for computer implementation.