{'fraction_non_alphanumeric': 0.028945056081046156, 'fraction_numerical': 0.011112834031115935, 'mean_word_length': 4.859479103573592, 'pattern_counts': {'":': 1, '<': 0, '<?xml version=': 0, '>': 4, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Rising Above Chaotic Likelihoods

Berliner (Likelihood and Bayesian prediction for chaotic systems, J. Am. Stat. Assoc. 1991) identified a number of difficulties in using the likelihood function within the Bayesian paradigm which arise both for state estimation and for parameter estimation of chaotic systems. Even when the equations of the system are given, he demonstrated"chaotic likelihood functions"both of initial conditions and of parameter values in the Logistic Map. Chaotic likelihood functions, while ultimately smooth, have such complicated small scale structure as to cast doubt on the possibility of identifying high likelihood states in practice. In this paper, the challenge of chaotic likelihoods is overcome by embedding the observations in a higher dimensional sequence-space; this allows good state estimation with finite computational power. An importance sampling approach is introduced, where Pseudo-orbit Data Assimilation is employed in the sequence-space, first to identify relevant pseudo-orbits and then relevant trajectories. Estimates are identified with likelihoods orders of magnitude higher than those previously identified in the examples given by Berliner. Pseudo-orbit Data Assimilation importance sampler exploits the information both from the model dynamics and from the observations. While sampling from the relevant prior (here, the natural measure) will, of course, eventually yield an accountable sample, given the realistic computational resource this traditional approach would provide no high likelihood points at all. While one of the challenges Berliner posed is overcome, his central conclusion is supported."Chaotic likelihood functions"for parameter estimation still pose a challenge; this fact helps clarify why physical scientists maintain a strong distinction between the initial condition uncertainty and parameter uncertainty.


Introduction
Nonlinear chaotic systems pose several challenges both for state estimation and for parameter estimation. Chaos as a phenomenon implies sensitive dependence on initial condition: initially nearby states will eventually diverge in the future. The bifurcations of various chaotic systems [20] reveal how the behavior of the system differs as a parameter value changes. One might think that likelihood and Bayesian analysis should be able to obtain good estimation both of initial conditions and of parameter values without much trouble. Berliner [2] examined the log-likelihood function of estimates of initial conditions and parameter values for the Logistic Map. He pointed out that chaotic systems can lead to "chaotic likelihood functions", suggesting that Bayesian analysis would require prohibitively intensive computing. The failure of variational approaches, when applied to long window observations of chaotic systems [6,8,14], supports his point. Sensitivity to initial condition also suggests that information in the observations (even over a relatively short range) can lead to good estimates of the initial condition [9]. An importance sampling approach for extracting such information without "intensive computing" is deployed in this paper. Adopting the Pseudo-orbit Data Assimilation (PDA) approach [6,8] recasts the task into a higher dimensional sequence space, where truly high likelihood states are successfully located near the trajectory manifold. Although statisticians often fail to make a strong distinction between initial conditions and parameter values, the challenges of initial condition estimation and parameter estimation are dissimilar for chaotic systems. PDA does not easily generalize to parameter estimation, as it is unclear how to mathematically define a relevant subspace of parameter space in which high likelihood trajectories might exist. Thus challenges remain in identifying high likelihood parameter values given the initial condition; this asymmetry is used to discuss differences between the initial conditions and parameter values. In terms of estimating initial conditions given the parameter values, however, Berliner's challenge is met and resolved without prohibitively intensive computing.

Chaotic Likelihood Function of Initial Conditions
Following Berliner [2] the Logistic Map is adopted as the system, assuming that the parameter a = 4 is known but the true initial statex 0 is not. In that case, the experiment is said to fall within the perfect model scenario 1 .The evolution of system states x i ∈ R m is then governed by the nonlinear dynamics f : Assuming additive observational noise δ i yields observations, s i =x i +δ i wherex is the true system state (Truth) and the observational noise, δ i , is Independent Normally Distributed (IND, δ i ∼ N(0, σ 2 )). Under this normality assumption, the log-likelihood (LLik) function is: where f i is the i th iteration of f , s i is the i th observation, and n is the duration of observations considered. Figure 1 shows the chaotic likelihood structure of 1024 samples from U(0, 1). Panel (a) plots the log-likelihood for x 0 , this can be contrasted with various panels in Berliner [2] 1 By assuming a perfect model and generating the data on a digital computer, one avoid the issue of "round-off error": the fitted model is evaluated on the same computer. For discussion, see [1,17,18] and references thereof. 2 For m = 1, the state x i is a scalar.
impact of making much more precise observations, consider a case wherex 0 is known to be within a region of radius only σ/10. Figure 2b shows the RLLik of 1024 uniformly sampled states in the region around the Truth with σ/10 radius. Yet again, no high likelihood state are identified. This difficulty here has nothing to do with the Likelihood approaches as there are high likelihood states other than Truth. One may demonstrate that such high likelihood states exist by sampling the points on a logarithmic spiral approaching the Truth (to machine precision) 6 . Figure 3 shows that other than Truth there exist high likelihood states, i.e. some of the blue points. A smooth curve of the log-likelihood function is only observed within a radius ofx 0 smaller than ∼ 10 −7 , see Figure 3b.
Without knowing the Truth, of course, this approach to identifying the blue points is inaccessible. The likelihood function is extremely jagged; as Berliner [2] stressed such chaotic likelihoods suggests that finding even one high likelihood state by sampling the state space would be prohibitively costly, making the approach inapplicable. That said there is no sense in which "sensitivity to the initial conditions" can be taken to imply that the information in the initial condition is "forgotten" or "lost". There is sufficient information in the observation segment to identify high likelihood initial states. Candidate states with non vanishing RLLik can be found by extracting the information from the system dynamics using a relatively new approach to data assimilation, which will be interpreted as an Importance Sampling.
3 Importance Sampling via Pseudo-orbit Data Assimilation 3

.1 Methodology
To locate high likelihood states, simply sampling in state space is inefficient. As the dimension of the system increases, this inefficiency makes the task computationally impractical. Importance sampling 7 (IS) locates high likelihood states in the trajectory manifold by adopting the Pseudo-orbit Data Assimilation approach [6,8]. PDA takes advantage of the known dynamics in a higher dimensional sequence space. A brief introduction of the PDA approach is given in the following paragraph (see [6,8] for additional details). 6 In this experiments 1024 points are generated byx 0 + 2 −(10+ 60i 1024 ) ǫ i , i = 1, 2, ..., 1024 where ǫ i is random drawn from U (0, 1). 7 In high-dimensional space the sampler targets the relevant low-dimensional trajectory manifold which is more efficient than sampling a hypersphere. Even in the one dimensional Logistic Map this approach succeeds by using PDA to sample the trajectory manifold in the n-dimensional sequence space.
Given a perfect dynamical model of dimension m, a perfect knowledge of the observational noise model, and a sequence of n observations s i , i = 0, ..., n − 1, define a sequence space as the m×n dimensional space consisting of all sequences of n states u i . 8 Most points in sequence space do not correspond to a trajectory of the system. Define a pseudo-orbit, U ≡ {u 0 , ..., u n−2 , u n−1 }, to be a point in the m × n dimensional sequence space for which u i+1 = f (u i ) for one or more components of U. Thus a pseudo-orbit corresponds to a sequence of system states which is not a trajectory of the system. Define the mismatch to be: By construction, system trajectories have a mismatch of zero. The mismatch cost function is then given by: Pseudo-orbit Data Assimilation minimizes the mismatch cost function for U in the m × n dimensional sequence space. If a gradient descent (GD) approach is adopted 9 , then a minimum of the mismatch cost function can be obtained by solving the ordinary differential where τ denotes algorithmic time. 10 A sequence of observations in the system state space define an initial pseudo-orbit, so called observation-based pseudo-orbit, S ≡ {s 0 , ..., s n−2 , s n−1 }, which with probability one will not be a trajectory. In practice, the minimization is initialized with the observation-based pseudo-orbit, i.e. 0 U = S where the pre-super-script 0 on U denotes the initial stage of the GD. The pseudo-orbit is a point in sequence space, 8 For the Logistic Map where m = 1, both s i and u i are scalars. 9 Other methods for this minimization are available, GD is discussed here due to its simplicity. 10 The approach can be generalized to situations where gradient of the model is not known analytically [10]; improving the ability to work without gradient information would widen the applications of the approach significantly. Logistic Map is a two-to-one map, and in most cases 12 only one of the two preimages for each α u i is relevant tox i−1 . In practice a criteria to discard irrelevant preimages must be defined, a simple example would be to discard (with high probability) those preimages whose distance from the corresponding previous observation exceeds some threshold based on the standard deviation of the observational noise (a 3σ criteria is used to generate the results presented in the following section).

Results
The green points in Figure 4 are located using the IS approach; Note that some have RLLik close to 0. As expected, the observations do not contain sufficient information to identify the state of the system at the time of the final observation with the same degree of precision. This is reflected in the fact that the green points are much less close to the true state at time 31 (Figure 4b) than those at time 0 (Figure 4a).
Two experiments were conducted to test the robustness of the IS approach. The first is 11 All points on the trajectory manifold have zero mismatch (are trajectories) and only points on the trajectory manifold have zero mismatch. 12 Not in all cases, however. For discussion of the point see [11]. The second is based on 2048 different true initial conditions to examine robustness. Three different observation window lengths were used in each experiment. Table 1 and Table 2 shows the results.
Given uncertain observations, one can never identify the Truth of a chaotic system unambiguously as was noted by Lalley [11,12] and later explored by Judd and Smith [8].
Using the IS approach, high likelihood states (IS states) are indeed found, as the states whose RLLik> −1 are found in every single experimental run. The fact that some IS states have greater likelihood than the Truth supports the expectation that the Truth is not expected to be the most likely system state given the observations.
For each experimental run, the minimum distance between those IS states (whose RLLik> −1) and the Truth is recorded. The minimum, maximum and median statistical values of the minimum distance from the Truth are reported in Table 1     statistics of the number of states (whose RLLik > 0) iii) statistics of the minimum distance between the states (whose RLLik > −1) and the Truth.
Complications arise from the fact that the Logistic Map is two-to-one; these have nothing to do with chaos per se (beyond the fact that one-to-one maps in one-dimension cannot display chaotic dynamics). Moving to higher dimensional 14 one-to-one maps, the calculation of preimages becomes straightforward. 13 Drawing samples uniformly from within a distance of 0.1 of Truth would require ∼ 10 8 candidates in order to find a candidate within ∼ 2 × 10 −10 of Truth. Such a procedure need not identify any high likelihood states. The results of Table 1 and 2 were obtained with only 1024 GD minimization iterations in each realization (each and every one of which identified high likelihood states close to Truth). 14 where the model state becomes a state vector The experiments above demonstrate that truly high likelihood points can be located using dynamical information. This eases Berliner's identification problem of initial condition with the appearance of chaotic likelihoods. Selecting an ensemble from this high likelihood set allows for informative forecasts which do not become useless until long after those from the point forecasts illustrated by Berliner [2] become uninformative.

Relative likelihoods
Maximum Likelihood Estimation has been widely used for estimation [15] since it was introduced by Fisher [7] in 1922. The "best" estimate is often chosen from a set of samples and only the relative likelihood in that sample is considered. Figure 5a shows the loglikelihood of 1024 states (the same set used in Figure 2b), the grey dashed line is the median log-likelihood of those states. In this case, it is not the problem of which estimate one shall pick, but how to show that they are all "bad" estimates. In practice, the Truth is unknown therefore it cannot be used as a reference like the cross plotted in Figure 2.
Given the observations and the noise model, however, the expected log-likelihood of the Truth 15 can be derived and serve as a reference. Figure 5b, the log-likelihood of 1024 states are plotted along with the expected log-likelihood of the Truth (black dashed line). Figure 5b shows that it is not a case of equifinality in Figure 5a but a case of equidismality.
In cases where it is observed that all traditional candidate states have vanishingly small log-likelihood relative to the expected log-likelihood of the Truth, approaches like those suggested above might prove valuable. 15 The log-likelihood of the Truth is , Z is a random variable following chi-squared distribution with n degrees of freedom. Statistics of the log-likelihood of the Truth can therefore simply derived from Z.  [2], there are fundamental differences in the information available to address these two distinct estimation problems.
Given the structure of the model class, the model parameter value determines the dynamical behaviour of the model (e.g. the natural measure) which is not changed by the initial condition. Given the model and its parameter value(s), the invariant measure constrains the relevant set(s) of initial conditions in the state space (and thereby trajectories in the sequence space). It is unclear how to construct similar constraints (if they exist 16 ) on the parameter values in the parameter space given the "true" initial state. Uncertainty in initial state differs from uncertainty in the parameter value. The information in a measurement of the initial condition uncertainty will decay with time and eventually becomes 16 It is not clear either how to construct the set of parameter values whose corresponding invariant measure contains the "true" initial state, or how to exploit this set, while it is clear how to exploit the existence of trajectory manifold given a particular value of the parameter. statistically indistinguishable from a random sample of the natural measure, while the information on each member from an ensemble under parameter uncertainty is preserved (and can be straight forwardly extracted given a trajectory segment), arguably forever.
While assuming the parameter value is perfect may not be ideal, it is not so nonsensical given that one has already assumed that the model structure is perfect. Assuming the initial state is perfect indicates a noise free observation is possible. Let the model's parameters be contained in the vector a ∈ R l . A set of l + 1 sequential noise free observations s i , s i+1 , ..., s i+l would, in general, be sufficient to determine a [13]. If one noise free observation is obtainable, obtaining only a few more noise free observations would define the true parameter value precisely. A more realistic way to put the problem is to estimate the parameter value(s) given the observations without assuming the "true" initial condition is known or even exists. In that case, the goal is to locate high likelihood trajectories (Smith et al. [19] call these shadowing trajectories) defined by the parameter values. Unfortunately it is not clear how to solve such a problem. In fact, it is not clear how to constrain the solution in the parameter space in a manner that reflects the constraints in the space of initial condition achieved by using trajectory manifold in the state space. Given a perfect model structure and knowing the true parameter value(s), the true initial state is a well defined goal of the identification. Inasmuch as structural model errors imply no true parameter value exists [5,9], it is unclear how one might define "true" initial state and the goal of estimation must be rethought.
Despite the importance of model parameters, there is no general method of parameter estimation outside linear systems. Methods have been developed to obtain useful parameter values with some success: McSharry and Smith [13,16] estimate model parameters by incorporating the global behaviour of the model into the selection criteria; Creveling et al. [3] have exploited synchronization for parameter estimation; Smith et al. [19] focused on the geometric properties of trajectories; Du and Smith [4] select parameter values based on the Ignorance Score of ensemble forecasts. Each of these methods, however, require a large set of observations. Challenges remain when only a short sequence of observations is available.
Berliner illustrated that even in the perfect model scenario traditional likelihood methods are unable to provide good estimates of the initial condition for nonlinear chaotic systems.
In large part, the failure is due to the inability of those approaches to skillfully meld the information in the dynamics of the nonlinear system itself with that in the observations. The importance sampling approach presented here better combines information from both observations and dynamics, thereby locating high likelihood initial states; this achieves an aim Berliner (1991) argued to be impossible by traditional methods. Despite the similarity of state estimation and parameter estimation, there are fundamental differences between uncertainty in the initial state and uncertainty in parameter value. Significant challenges remain in solving the challenge chaotic likelihood functions pose in parameter estimation.