{'fraction_non_alphanumeric': 0.04482206624799902, 'fraction_numerical': 0.011184993637893527, 'mean_word_length': 4.323609745438654, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 41, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Qualitative Robustness in Bayesian Inference

The practical implementation of Bayesian inference requires numerical approximation when closed-form expressions are not available. What types of accuracy (convergence) of the numerical approximations guarantee robustness and what types do not? In particular, is the recursive application of Bayes' rule robust when subsequent data or posteriors are approximated? When the prior is the push forward of a distribution by the map induced by the solution of a PDE, in which norm should that solution be approximated? Motivated by such questions, we investigate the sensitivity of the distribution of posterior distributions (i.e. posterior distribution-valued random variables, randomized through the data) with respect to perturbations of the prior and data generating distributions in the limit when the number of data points grows towards infinity.


Introduction
Robust Bayesian Analysis [6,8] seeks to quantify variations in the output of Bayesian inference with respect to perturbations of the model and the prior. The importance of such an effort is expressed by Wasserman, Lavine and Wolpert [87]: "Most statisticians would acknowledge that an analysis is not complete unless the sensitivity of the conclusions to the assumptions is investigated. Yet, in practice, such sensitivity analyses are rarely used. This is because sensitivity analyses involve difficult computations that must often be tailored to the specific problem. This is especially true in Bayesian inference where the computations are already quite difficult." In the classical Robust Bayesian framework, one defines a model and a prior and describes a perturbation class for the model and the prior and then asks, given the specific value of the data, what is the range of possible conclusions (posterior values) as the model and the prior varies over their perturbation classes. Although robustness has been established for classes of finite dimensional perturbations [6,8,11,89], recent results [69,68], summarized in [70], suggest that, under finite information, that is, finite codimensional perturbations (e.g., when the class of perturbed priors are defined via TV/Prokhorov metrics or a finite number of constraints on marginal distributions), Bayesian inference is not only not robust, it is brittle, that is, extremely non-robust.
The mechanism generating this brittleness can be seen as an extreme occurrence of the dilation phenomenon [88] which, in robust Bayesian inference, refers to the enlargement of optimal bounds caused by the data dependence of worst priors. This data dependence of worst priors is inherent to the formulation of classical Bayesian Sensitivity Analysis, in which worst priors are computed given the specific value of the data.
Therefore, although the brittleness results of [69,68] suggest that Bayesian inference may not be robust under finite information within the classical Robust Bayesian framework [69,68], one may ask whether robustness could be established under finite information by exiting this strict framework and computing the sensitivity of posterior conclusions independently of the specific value of the data. To investigate this question, this paper will generalize Hampel's [45] notion of qualitative robustness to Bayesian inference based on the quantification of the sensitivity of the distribution of posterior distributions with respect to perturbations of the prior and the data generating distribution, in the limit when the number of data points grows towards infinity. Note that, contrary to classical Bayesian Sensitivity Analysis, in the proposed formulation, the data is not fixed and posterior values are therefore analyzed as dynamical systems randomized through the distribution of the data.
2 Structure of the paper and non-robustness mechanisms 2

.1 General set up
We will now describe the general setup allowing us to analyze posterior distributions as measure-valued random variables or measure-valued dynamical systems randomized through the distribution of the data. We defer measure theoretic technicalities until later. Let Θ and X be measurable spaces. We write M(X) for the set of probability distributions on X, M(Θ) for the set of probability distributions on Θ and M 2 (Θ) := M M(Θ) the set of probability distributions on the set of probability distributions on Θ. We fix a prior π ∈ M(Θ), a model P : Θ → M(X), and consider a data generating distribution µ ∈ M(X). Let µ n ∈ M n (X) denote the n-fold product measure corresponding to taking n-i.i.d. samples and for any such n-sample x n from µ n , let π x n ∈ M(Θ) denote the posterior measure associated with the prior, model, and the sample. Then we ask how these posteriors π x n vary as a function of the sample data x n when it is generated by i.i.d. sampling from µ, that is x n ∼ µ n . To do so, we consider the mapπ : X n → M(Θ) defined by the determination of the posteriors π(x n ) := π x n (2.1) and consider its corresponding pushforward operator where we have removed the bar over π in the notation to remind us that this pushforward operator π * corresponds to the prior π. Note that π * µ n is the distribution of the posterior distribution π x n (of the parameter θ) when the data x n is a random variable distributed according to µ n . Since M n (X) ⊂ M(X n ) and µ n ∈ M n (X), it follows that µ n ∈ M(X n ) so that the pushforward π * µ n ∈ M 2 (Θ) (2.2) is well-defined. In Section 4 we will define the qualitative robustness of Bayesian inference as the property that π * µ n and π * (µ ) n can be made arbitrarily close for sufficiently large n if the priors π and π and the data generating distributions µ and µ are close enough. Most of the non-robustness mechanisms presented in this paper only require perturbations in the prior distributions (that is, µ = µ ) and so are particularly relevant to well-specified Bayesian inference. Of course the notion of qualitative robustness will depend on the metrics placed on the space M(X) of data-generating distributions, the space M(Θ) of prior distributions on Θ and the space M 2 (Θ) of random distributions on Θ. The total variation, Prokhorov and Ky Fan metrics will be of particular interest in our analysis.

Robustness and Consistency
Since the proposed notion of qualitative robustness is established in the limit when the number of data points grows towards infinity, it is natural to expect that the notion of consistency (i.e., the property that posterior distributions convergence towards the data generating distribution) will play an important role.
Although consistency is primarily a frequentist notion, according to Blackwell and Dubins [9] and Diaconis and Freedman [19], consistency is equivalent to intersubjective agreement which means that two Bayesians will ultimately have very close predictive distributions. Therefore, it also has importance for Bayesians. Fortunately, not only are there mild conditions which guarantee consistency, but the Bernstein-von-Mises theorem goes further in providing mild conditions under which the posterior is asymptotically normal. The most famous of these are Doob [21], Le Cam and Schwartz [60], and Schwartz [78,Thm. 6.1]. For more recent developments, see Barron, Schervish and Wasserman [3], Barron [4], Wasserman [86], Ghosh, Ghosal and Ramamoorthi [35], Ghosh [38], Kleijn [56], Walker [84], and the excellent review by Ghosal [34]. Moreover, the assumptions needed for this consistency are so mild that one can be lead to the conclusion that the prior does not really matter once there is enough data. For example, we quote Edwards, Lindeman and Savage [27]: "Frequently, the data so completely control your posterior opinion that there is no practical need to attend to the details of your prior opinion." On the other hand, seemingly paradoxical results regarding inconsistency were found by Diaconis and Freedman [19,29], and Freedman [28,30,31]: one consequence of Freedman [31] is that the set of pairs of priors leading to concurring asymptotic posterior conclusions is meager, i.e. very small in a topological sense.
Moreover, it appears that the full implications of model misspecification, when the data generating distribution is not exactly in the model class, have only been appreciated somewhat recently, see, e.g. Kleijn [55], Grünwald [42], Grünwald and Langford [41], Müller [65], Kleijn and van der Vaart [57,58], and applied investigations in Biology begun in Douady et al. [22] and in Economics in Lubik and Schorfheide [61]. We quote from Kleijn and van der Vaart [58]: "In the misspecified situation the posterior distribution of a parameter shrinks to the point within the model at minimum Kullback-Leibler divergence to the true distribution, a consistency property that it shares with the maximum likelihood estimator. Consequently one can consider both the Bayesian procedure and the maximum likelihood estimator as estimates of this minimum Kullback-Leibler point. A confidence region for this minimum Kullback-Leibler point can be built around the maximum likelihood estimator based on its asymptotic normal distribution, involving the sandwich covariance. One might also hope that a Bayesian credible set automatically yields a valid confidence set for the minimum Kullback-Leibler point. However, the misspecified Bernstein-Von Mises theorem shows the latter to be false." That is, in the misspecified case, not only do we have inconsistency, we also have strong convergence -with consequences for confidence sets. Indeed even earlier, to some, the consistency results appeared to generate more confidence than possibly they should. We quote A. W. F. Edwards [26,Pg. 60]: "It is sometimes said, in defence of the Bayesian concept, that the choice of prior distribution is unimportant in practice, because it hardly influences the posterior distribution at all when there are moderate amounts of data. The less said about this 'defence' the better." We will demonstrate that the Edwards defence is essentially what produces the non qualitative robustness of Bayesian inference. In particular, the assumptions required for consistency are such that arbitrarily small local perturbations of prior distributions (near the data generating distribution) result in consistency or non-consistency, and therefore, have large impacts on the asymptotic behavior of posterior distributions. To make this precise, in the following two sections we develop the notions of consistency and robustness of Bayesian inference. In particular, in Section 3 we develop the consistency theorem of Schwartz so that it can easily be used in the robustness analysis that is developed Section 4. Such a relationship between robustness and consistency has been observed recently in Hable and Christmann [44] in ill-posed classification and regression problems and Owhadi et al. [69,70]. Moreover, with extra work and stronger but still mild assumptions, rates for consistency can be obtained, see e.g. Shen and Wasserman [79], Ghosal, Ghosh, and van der Vaart [37], Huang [47], and in the misspecified case Kleijn and van der Vaart [58]. In particular, the availability of local conditions guaranteeing rates to consistency has been investigated in Martin, Hong and Walker [63]. We conjecture that when these results are applicable, that they can be used to increase the degree of non qualitative robustness of Bayesian inference in a quantitative way. The data generating distribution is P (θ * ). π has most of its mass around θ. π is an arbitrary small perturbation of π so that π has Kullback-Leibler support at θ * . Corollary 3.1 implies that π n converges towards δ θ * while π n remains close to δ θ .

Mechanisms generating non-robustness
For the clarity of the paper, in this subsection, we introduce some of the mechanisms generating the non qualitative robustness of Bayesian inference. More precisely, the first illustrations presented are simplified graphical representations of the mechanisms used to obtain the main (non qualitative robustness) results of Section 5 which do not utilize any misspecification. Then we present mechanisms which appear to generate non qualitative robustness due to misspecification.
The core mechanism is derived from the nature of both the assumptions and assertions of results supporting consistency. A prototypical example of such results, which we will use in our proofs, is the corollary to Schwartz' consistency theorem presented in Section 3. More precisely, using the notations of Section 2.1, Corollary 3.1 states that if the data generating distribution is µ = P (θ * ) and if the prior π attributes positive mass to every Kullback-Leibler neighborhood of θ * ∈ Θ then the posterior distribution converges towards δ θ * as n → ∞. The assumption that π attributes positive mass to every Kullback-Leibler neighborhood of θ * ∈ Θ does not require π to place a significant amount of mass around θ * , but instead can be satisfied with an arbitrarily small amount. Therefore, if, as in Figure 2.1, π is a prior distribution with support centered around θ = θ * , but with a very small amount of mass about θ * , so that it satisfies the assumptions of Corollary 3.1 at θ * , then π can be slighted perturbed into a π with support also centered around θ = θ * , but with no mass about θ * . In this situation, although π and π can be made arbitrarily close in total variation distance, the posterior distribution of π converges towards δ θ * as n → ∞, whereas that of π remains close to δ θ . Figure 2.2 gives an illustration of the same phenomenon when the parameter space Θ is the interval [0, 1] and the probability density functions of π and π are p and p with respect to the uniform measure.
Note that the mechanism illustrated in Figures 2.1 and 2.2 does not generate non qualitative robustness at all priors but instead for the full class of consistency priors, defined by the assumption of having positive mass on every Kullback-Leibler neighborhood  The data generating distribution is P (θ * ). The probability density functions of π and π are p and p with respect to the uniform distribution on [0, 1]. π is an arbitrarily small perturbation of π in total variation. π n converges towards δ θ * while the distance between the support of π n and θ * remains bounded from below by a > 0.
of θ * . One may wonder whether this non qualitative robustness can be avoided by selecting the prior π to satisfy Cromwell's rule (that is, the assumption that π gives strictly positive mass to every nontrivial open subset of the parameter space Θ). Theorem 5.4 shows that this is not the case if the parameter space Θ is not totally bounded. For example, when Θ = R, for all δ > 0 one can find θ ∈ R such that the mass that π places on the ball of center θ and radius one is smaller than δ, and by displacing this small amount of mass one obtains a perturbed prior π whose posterior distribution remains asymptotically bounded away from that of π when the data-generating distributions is P (θ). Similarly if Θ is totally bounded then Theorem 5.5 places an upper bound on the size of the perturbation of the prior π that would be required as a function of the covering complexity of Θ. Note that these observations suggest that a maximally qualitatively robust prior should place as much mass as possible near all possible candidates θ for the parameter θ * of the data generating distribution, thereby reinforcing the notion that a maximally robust prior should have its mass spread as uniformly as possible over the parameter space. We refer to Section 6 for a discussion on the existence of such measures in relation to the geometry of the sets of measures having local and global Kullback-Leibler support.

Robustness under misspecification
Although the main results of Section 5 do not utilize any model misspecification, the brittleness results of [69] suggest that misspecification should also generate non qualitative robustness. Indeed, although, one may find a prior that is both consistent and qualitatively robust when Θ is totally bounded and the model is well-specified, we now show how extensions of the mechanism illustrated in Figures 2.1 and 2.2 suggest that consistency implies non qualitative robustness under misspecification. Consider the example illustrated in Figure 2.3. In this example the model P is the  We assume that the model P : Θ 1 → M(X) is the restriction of an injective modelP : Θ 1 × Θ 2 → M(X) to Θ 1 × {θ 2 = 0}. The data generating distribution is P (θ * ) where θ * := (θ * 1 , θ * 2 ), with θ * 2 = 0, so that the model P is misspecified. π satisfies Cromwell's rule. π is an arbitrarily small perturbation of π having Kullback-Leibler support at θ * . Corollary 3.1 implies that π n converges towards δ θ * while the distance between the support of π n and θ * remains bounded from below by a > 0.  The data generating distribution is µ ∈ P (Θ), so that the model is misspecified. ν is an arbitrarily small perturbation of P * π in total variation distance having non-zero mass on µ. Note that if a small mass on µ is sufficient to ensure the consistency of ν then such a mechanism also implies the non-robustness of the model with respect to misspecification since in such a case, ν n , the posterior distribution of ν would converge towards δ µ whereas the distance between the support of P * π n and µ remains bounded from below by the distance from the model to the data generating distribution.
restriction of a well specified larger modelP : Θ 1 × Θ 2 → M(X) to θ 2 = 0. Assume that the data generating distribution isP (θ * 1 , θ * 2 ) where θ * 2 = 0, so that the restricted model P is misspecified. Let π be any prior distribution on Θ 1 × {θ 2 = 0}. Although π may satisfy Cromwell's rule the mechanisms presented in this paper suggest that is not qualitatively robust with respect to perturbed priors having support on Θ 1 × Θ 2 . Indeed, let π be an arbitrarily small perturbation of π obtained by removing some mass from the support of π and adding that mass around θ * . Note that π can be chosen arbitrarily close to π while satisfying the local consistency assumption of Corollary 3.1 which implies that the posterior distributions of π concentrate on θ * while the posterior distributions of π remain supported on Θ 1 ×{θ 2 = 0}. Note that ifP is interpreted as an extension of the model P , then this mechanism suggests that we can establish conditions under which Bayesian inference is not qualitatively robust under model extension. Figure 2.4 represents a non-parametric generalization of the mechanism of Figure  2.3. Assume that the data generating distribution is µ ∈ P (Θ), so that the model is misspecified. Let π ∈ M(Θ) be an arbitrary prior distribution and P * π ∈ M 2 (X) its corresponding non-parametric prior. By removing an arbitrarily small amount of mass from P * π and placing it on µ one obtains an arbitrarily close prior distribution ν that is consistent with respect to the data generating distribution µ. Therefore although P * π and ν may be made arbitrarily close, their posterior distributions would remain asymptotically separated by a distance corresponding to the degree of misspecification of the model (the distance from µ to P (Θ)).

Robustness and Computational Bayesian Inference
One of the reasons behind Bayesian inference's recent surge in popularity is the availability of computational methodologies and environments to compute the posteriors, such as Markov chain Monte Carlo (MCMC) simulations. Indeed, Diaconis' [18] "Markov chain Monte Carlo revolution", indicates that MCMC is a powerful tool to implement Bayesian inference. More generally, see Neal [67] for a review of the use of MCMC in inference.
If we consider the computation of Bayesian inference in the context of a Machine Learning algorithm then the practical effects of computation need to be incorporated; both the computational requirements and the impact on performance and qualitative robustness. Indeed, when posterior distributions are approximated using MCMC simulations, the robustness analysis naturally includes not only quantifying sensitivities with respect to the choice of prior but also the analysis of convergence and stability of the computational method. This is particularly true in Bayesian updating where Bayes' rule is applied iteratively and computed/approximated posterior distributions are then treated as prior distributions. The singular, and apparently antagonistic, relationship between qualitative robustness and consistency presented in this paper suggests that the metrics used to analyze convergence and qualitative robustness should be chosen with care and not independently from each other.
Originally, the convergence of MCMC has been generally analyzed in terms of total variation, see e.g. Roberts and Rosenthal [74,75]. However, as cautioned by Gelman [33], although MCMC is a powerful tool for Bayesian inference, it is also "so easy to apply that there are risks of serious errors." One such risk appears to be related to the absence of analysis: According to Roberts and Rosenthal [74, Pg. 10] "rigorous quantitative bounds are not available in general" and more recently Madras and Sezer [62] assert that "quantitative rigorous results about realistic examples are scarce". Possibly to rectify this situation, recently the Wasserstein metric is also considered, see e.g. Gibbs [39] and Madras and Sezer [62]. For the definition see Dudley [24]. For a separable metric space, according to Zolotarev [ Therefore, on Polish metric spaces of bounded diameter, the Wasserstein metric is equivalent to the Prokhorov metric and therefore metrizes weak convergence. Now let us incorporate the MCMC analysis into the robustness analysis when the posteriors are computed using MCMC simulations. Let us further suppose that we are in a situation where we can prove when the computation will have converged, or prove that it has converged using diagnostics such as found in Carlin and Chib [12] and Cowles and Carlin [15]. It then appears that incorporating the MCMC simulation into the robustness analysis makes establishing robustness in any topology stronger than that which we can establish the convergence of the MCMC appear problematic. Consequently, it appears that establishing qualitative robustness of Bayesian inference in the total variation topology on M(Θ) might be too much to ask, while weaker topologies, such as the Wasserstein or Prokhorov topologies, might facilitate practical robustness analysis for important problems.

Lorraine Schwartz' Theorem
As described in Section 2.2, robustness and consistency are closely related properties and consistency will be at the core of the mechanism generating non-robustness. The breakthrough in consistency for Bayesian inference is considered to be Schwartz' theorem [ Recall the Kullback-Leibler divergence K between two measures µ 1 and µ 2 defined by where ν is any measure such that both µ 1 and µ 2 are absolutely continuous with respect to ν. It is well known that K is nonnegative, and that it is finite only if µ 1 µ 2 , and in that case For a model P : Θ → M(X), there is the pullback to a function K on Θ defined by K(θ 1 , θ 2 ) := K(P θ 1 , P θ 2 ) and when the model is dominated by a σ-finite measure ν, if we let p(x|θ) := dP θ dν (x), x ∈ X be a realization of the Radon-Nikodym derivative, then the pullback has the form From this we define a Kullback-Leibler neighborhood of a point θ ∈ Θ by Let us define the set of priors K(θ) ⊂ M(Θ) which have Kullback-Leibler support at θ by which implicitly requires that K (θ) be measurable 1 for all > 0. Also let K ⊂ M(Θ) denote those measures with global Kullback-Leibler support, that is, is the set of priors which have Kullback-Leibler support at all θ, and let K ae ⊃ K, defined by denote the set of priors with almost global Kullback-Leibler support. Let us address the measurability of the Kullback-Leibler neighborhoods K (θ) ⊂ Θ, > 0. For the nonparametric case, Barron, Schervish and Wasserman [3,Lem. 11] demonstrate that the Kullback-Leibler neighborhoods K (P θ * ) ⊂ M(X) are measurable with respect to the strong topology restricted to the subspace of measures which are absolutely continuous with respect to a common σ-finite reference measure. For the parametric case, Dupuis and Ellis [25,Lem. 1.4.3] assert that on a Polish space that K is lower semicontinuous in both arguments. Since the subset embedding : X → X of a subset X of a metric space X is isometric, when X is a Borel subset of a separable metric space X , it can be shown that the induced pushforward map i * : M(X) → M(X ) is isometric in the Prokhorov metrics, in particular it is continuous. Since the composition of a continuous and a lower semicontinuous function is lower semicontinuous, it follows from Dupuis and Ellis [25,Lem. 1.4.3] that on any realization of a standard Borel space that the Kullback-Leibler divergence is lower semicontinuous in each of its arguments separately, in particular, fixing the first, it is lower semicontinuous. Therefore K (P θ * ) ⊂ M(X) is closed, and therefore measurable for > 0. Consequently, when P is measurable, it follows that K (θ * ) ⊂ Θ is measurable for > 0.
The following corollary to Schwartz' Theorem, and its implications in Proposition 3.4, gives us the form of consistency that we will use in the robustness analysis. Since it assumes the model P : Θ → M(X) is measurable, and since by Aliprantis and Border [1,Thm. 15.13] the map M(X) → R defined by µ → µ(A) is Borel measurable for all A ∈ B(X), it follows that P corresponds to a Markov kernel. Moreover since it also assumes the model to be dominated, it follows from Barron, Schervish and Wasserman [3,Lem. 10] that the Radon-Nikodym derivatives can be chosen so that they are B(X) × B(Θ) measurable. Consequently, for a prior π, such a choice determines a well-defined conditional measure π x n for any n-sample x n ∈ X n . Note the assumption that the map P : Θ → P (Θ) be open. Then for every π ∈ M(Θ) with Kullback-Leibler support at θ * ∈ Θ, for every measurable neighborhood U of θ * , we have it follows that P (Θ) is countable and therefore measurable. It also follows that Θ is countable, although it may not be discrete. Since P −1 (A) is countable, and therefore measurable, for all measurable A, it follows that P is measurable. Consequently, in this case the measurability and openness conditions on the model P of Theorem 5.1 follow from the assumption that P (Θ) ⊂ M(X) be discrete.
It will be useful to express the assertion of Corollary 3.1 and some of its consequences in terms of the convergence of measures and random measures. To that end, recall the notation M 2 (Θ) := M(M(Θ)), and consider the corresponding sequence of random variables π n : (X ∞ , P ∞ θ * ) → M(Θ), defined by π n (x ∞ ) := π x n , x ∞ ∈ X ∞ , and its induced sequence of laws (π n ) * P ∞ θ * ∈ M 2 (Θ). Note especially that δ δ θ * is the Dirac mass in M 2 (Θ) situated at the Dirac mass δ θ * in M(Θ) situated at θ * .

4)
where → is weak convergence. This in turn implies that

5)
for > 0, which is equivalent to

6)
where d P rr is the Prokhorov metric on M 2 (Θ) defined with respect to the Prokhorov metric d P r on M(Θ).

Qualitative Robustness of Bayesian Inference
Hable and Christmann [43] have recently established qualitative robustness for support vector machines. Hampel [45] introduced the notion of the qualitative robustness of a sequence of estimators and Cuevas [17] has extended Hampel's definition and his basic structural results to Polish parameter spaces. Boente et al. [10] have developed qualitative robustness for stochastic processes and Nasser et al. [66] for n large enough.
Of course, there are many variations on this theme, and the choice of metrics will affect not only the attainability of results, but the relevance of any results obtained. Metrics on spaces of measures is a well studied field, see e.g. Rachev et al. [73] and Gibbs and Su [40], but to keep the presentation simple, here we will restrict our attention to the total variation and Prokhorov metrics. On general measurable spaces, we can metrize the space of measures M(X) and the space of measures on the space of measures M 2 (Θ) using total variation. However, when X is metric, we can also metrize the space of measures M(X) with the Prokhorov metric. In the same way, when Θ is a metric space, we can metrize the space of measures M(Θ) also with the Prokhorov metric, and having metrized in any way, we can then proceed to metrize M 2 (Θ) using the Prokhorov metric. Using this setup with Polish spaces and the resulting Prokhorov metrics, Cuevas [17] proves in two cases, that common Bayesian models with specific fixed priors are not qualitatively robust.

Varying the prior and fixing the data generating distribution
Regarding the importance of the robustness of Bayesian inference with respect to the prior, we quote from Berger's [7] discussion on Diaconis and Ylvisaker [20]: "There is a very serious issue concerning such an approximation, however, namely the issue of whether this good approximation to the prior ensures that the posterior will also be well approximated. I think the answer, in general, is no." The stability of Bayesian decision theory with respect to the prior was fully initiated by Kadane and Chuang [50,51] and further developed in Chuang [13] and Salinetti [76], and positive results obtained. In particular, in [50] comparison with previous notions, in particular that of Edwards, Lindman and Savage [27], is made. In Kadane and Srinivasan [49,52] sufficient conditions for the stability of Bayes decision problems under uniform convergence of losses are obtained, generalizing the previously mentioned works [50,13,76].
However, we would like to proceed along the lines of Hampel's approach here, so that we can combine it with Section 4.1 to obtain a framework for qualitative robustness for Bayesian inference which simultaneously includes variation in the prior and the data generating distribution into one Hampel-like framework in a natural way. This has the added advantage that we can utilize the fundamental results of Hampel [45] and Cuevas [17], and points to further development which will be useful. For example, Mizera [64] has fully developed these notions of qualitative robustness to include both ill-definedness and multi-valuedness, an extension which would be extremely useful for any treatment of Bayesian inference which incorporates conditioning only being determined almost everywhere. To that end, let us now develop a definition of qualitative robustness with respect to variation in the prior, for fixed data generating distribution, based on Hampel.
For a single sample x ∈ X, Basu et al. [5] say that the Bayesian inference is qualitatively robust at π and x ∈ X, with respect to a metric d M(Θ) on M(Θ) and an admissible set Π ⊂ M(Θ) containing π, if given > 0, there exists a δ > 0 such that They provide many positive results and some negative results regarding the Prokhorov and Levy metrics. We can extend this definition easily to a sequence x ∞ := x i , i = 1, ..., as follows: for the first n, we let x n := {x i , i = 1, .., n} denote the n-sample and then define the sequence of posteriors π x n ∈ M(Θ), n = 1, ... Then we say that the Bayesian inference is qualitatively robust at π and x ∞ ∈ X ∞ using the notion of stability of a dynamical system: if, given > 0, there exists a δ > 0 such that π ∈ Π, d M(Θ) (π,π) < δ =⇒ d M(Θ) (π x n ,π x n ) < , for n large enough. However, this definition puts no distributional requirements on the sequence x ∞ . For an i.i.d. sample sequence x ∞ ∼ µ ∞ , we can include the "i.i.d. with respect to µ" assumption into the definition in natural way by saying that the Bayesian inference is qualitatively robust at π if given 1 , 2 > 0, there exists a δ > 0 such that π ∈ Π, d M(Θ) (π,π) < δ =⇒ µ n x n : d M(Θ) (π x n ,π x n ) > 1 < 2 for large enough n. This definition can be calibrated with the following single parameter version: given > 0, there exists a δ > 0 such that for large enough n, where by "calibrated" we mean that if µ n x n : d M(Θ) (π x n ,π x n ) > 1 < 2 , then if we define := max ( 1 , 2 ), we have so that we conclude that µ n x n : d M(Θ) (π x n ,π x n ) > < , and conversely, if µ n x n : d M(Θ) (π x n ,π x n ) > < , then µ n x n : d M(Θ) (π x n ,π x n ) > 1 < 2 with 1 := , 2 := .
We now express the condition (4.1) as convergence in probability of M(Θ)-valued random variables and metrize this convergence using the Ky Fan metric. To that end, consider the sequence of maps Then for every µ ∈ M(X), we use the same symbol π n to denote the corresponding sequence π n : (X ∞ , µ ∞ ) → M(Θ) of M(Θ)-valued random variables defined on the probability space (X ∞ , µ ∞ ).
To metrize the condition (4.1) as convergence of the random variables π n using the Ky Fan metric, recall that for a metric spaces S, the metric d : S ×S → R is a continuous function and therefore Borel measurable with respect to the Borel σ-algebra B(S × S). To proceed, suppose that the metric space M(Θ), d M(Θ) is separable. In particular, note that when Θ is a separable metric space, such as a Borel subset of a Polish metric space, then the metric space M(Θ), d P r , where d P r is the Prokhorov metric, is separable. Then for fixed data generating measure µ and two priors π,π ∈ M(Θ), the identity implies that the inequality µ n x n : d M(Θ) (π x n ,π x n ) > < on the righthand side of (4.1) can be written µ ∞ d M(Θ) (π n ,π n ) > < which, from the definition (4.2), implies that where α µ denotes the Ky Fan metric defined on the space of M(Θ)-valued random variables W : (X ∞ , µ ∞ ) → M(Θ) on the probability space (X ∞ , µ ∞ ).
Since, for fixed data generating measure µ, the sequence of random variables π n , n = 1, .. are all defined on the same probability space (X ∞ , µ ∞ ), the definition 4.1 of qualitative robustness now can be stated in terms of the Ky Fan metric on the space of M(Θ)-valued random variables.
for large enough n, where α µ is the Ky Fan metric on the space of M(Θ)-valued random variables on the probability space (X ∞ , µ ∞ ).
This seems to be the most reasonable generalization of Basu et al. [5] to i.i.d sequences. Of course, other definitions could be used -we simply must specify a metric α µ on M(Θ)-valued random variables.

Qualitative Robustness of Bayesian Inference -Definition
Now we take the ideas of the previous two subsections and combine them to allow the variation in both the prior and the data generating distribution. Recall from Section 4.1 that when the prior π is fixed and we vary the data generating distribution µ, we define a mapπ : X n → M(Θ) byπ (x n ) := π x n (4.4) and use the corresponding pushforward operator to pushforward µ n to π * µ n ∈ M 2 (Θ) .
Then we say that the Bayesian inference for prior π ∈ M(Θ) is qualitatively robust at µ with respect to an admissible set P containing µ, and metrics d M(X) and d M 2 (Θ) , if for any > 0, there exists a δ > 0 such that for n large enough. On the other hand, when the data generating distribution µ is fixed and we vary the prior π, we consider the sequence of maps and, for fixed µ, the resulting sequence of M(Θ)-valued random variable. Then, according to Definition 4.2, for an admissible set Π containing π, we say that the Bayesian inference is qualitatively robust if given > 0, there exists a δ > 0 such that π ∈ Π, d M(Θ) (π,π) < δ =⇒ α µ (π n ,π n ) < for large enough n, where α µ is the Ky Fan metric on the space of M(Θ)-valued random variables on the probability space (X ∞ , µ ∞ ). Of course, this definition also makes sense for other metrics α µ on that space of random variables. These two definitions can be combined in a straightforward manner to define robustness corresponding to a single prior/data generating pair. However, to define qualitative robustness for a larger class of distributions than a single pair, we let Z ⊂ M(Θ) × M(X) 2 denote the admissible set of prior-data generating distribution pairs Then the Bayesian inference is qualitatively robust with respect to Z, if given 1 , 2 > 0, there exists δ 1 , δ 2 > 0 such that for large enough n.

Proprokhorov Robustness
Consider the Definition 4.3 of qualitative robustness using the Ky Fan metric. As mentioned before the Ky Fan metric only makes sense when the metric space M(Θ) is separable. When Θ is is a separable metric space, such as a Borel subset of a Polish metric space, the space M(Θ) equipped with the Prokhorov metric d P r is separable. Consequently we now fix it to be (M(Θ), d P r ). Let us call Definition 4.3 using the Ky Fan metric defined on the space of (M(Θ), d P r )-valued random variables Ky Fan-Prokhorov robustness. We now define a weaker definition of qualitative robustness which we call Proprokhorov robustness such that Ky Fan-Prokhorov robustness implies Proprokhorov robustness. Consequently, and most importantly, Proprokhorov non-robustness implies Ky Fan-Prokhorov non-robustness. This weaker robustness has two distinct advantages. The first is that it has a simpler expression than Ky Fan-Prokhorov robustness and the second is that it is simpler to analyze. This simpler structure amounts to the Prokhorov metric on the space of probability measures on the space of probability measures equipped with the Prokhorov metric, suggesting the name Prokhorov-Prokhorov robustness, which we have shortened to Proprokhorov. Basic results which we will need regarding this metric space are derived in the appendix, Section 7.
Since it is essential for our main result, we summarize.

Main Results
Now that we have defined qualitative robustness for Bayesian inference and presented the consistency conditions of Section 3, we are now prepared for our main results. Although the brittleness results of [69,68] and the non qualitative robustness results of Cuevas [17] suggest that we may obtain non qualitative robustness according to Definition 4.3 by fixing the prior and varying the data generating distribution, here we instead fix the data generating distribution to be in the model class, and show that the inference is not Proprokhorov robust according to Definition 4.4, which by Theorem 4.5 implies that it is not Ky Fan-Prokhorov robust according to Definition 4.3, when the Ky Fan metric is defined on (M(Θ), d P r )-valued random variables. In particular, these results do not utilize any misspecification.
To that end, for θ ∈ Θ, let us denote Our Main Theorem shows, under the conditions of Schwartz' Corollary, that the Bayesian inference is not robust under the assumption that the prior has Kullback-Leibler support at the parameter value generating the data.
Remark 5.2. Actually the proof shows more; let D denote the diameter of Θ, then for < min ( D 2 , 1), there does not exist a δ > 0 such that the Definition 4.4 of Proprokhorov robustness is satisfied. Since min ( D 2 , 1) is large, either half the diameter of the space or larger than 1, we say the inference is brittle. Remark 5.3. In particular, Theorem 4.5 implies that the inference is not Ky Fan-Prokhorov robust.
Theorem 5.1 does not assert that the Bayesian inference is not robust at any specified prior, only that it is not robust under the assumption that the prior has Kullback-Leibler support at the parameter value generating the data. To establish non-robustness at specific priors we include variation in the data-generating distribution in the model class as follows. Let ∆ P ⊂ M(X) × M(X), defined by denote the fact that we allow the data generating distribution to vary throughout the model class but do not allow any perturbations to it. Then, for π ∈ M(Θ), define the admissible set Z ρ (π) ⊂ M(Θ) × M(X) 2 by where B tv ρ (π) is the open ball in the total variation metric. Since the following theorem is a corollary to the theorem after it, Theorem 5.5, we do not include its proof. However, we state it here because it is the more fundamental result.
Theorem 5.4. Given the conditions of Schwartz' Corollary 3.1 with Θ not totally bounded. Then if the prior π has Kullback-Leibler support for all θ ∈ Θ, the Bayesian inference is not Proprokhorov robust at Z ρ (π) for all ρ > 0.
Since a metric space is totally bounded if and only if its completion is compact, when Θ is totally bounded, we assume that it is a Borel subset of a compact metric space. In this case, although Theorem 5.4 does not apply, utilizing the covering number and packing number inequalities of Kolmogorov and Tikhomirov [59], we can provide a natural quantification of qualitative robustness. To that end, we define covering and packing numbers. For a finite subset Θ ⊂ Θ, the finite collection of open balls {B (θ), θ ∈ Θ } is said to constitute a covering of Θ if Θ ⊂ ∪ θ∈Θ B (θ). For a finite set Θ we denote its size by |Θ |. The covering numbers are defined by If the Bayesian inference is Proprokhorov robust, then given > 0, in terms of the product metric we must have , ρ .
Remark 5.6. Although the total variation metric on M(Θ) is not used in the metric defining qualitative robustness for separability, measurability, and consistency purposes, the definition of the admissible sets Z ρ (θ) and Z ρ (π) in terms of the total variation metric and a look at the proofs, can be used to show that these non qualitative robustness results primarily depend on total variation and do not arise because of the use of the weak topology.

Kullback-Leibler Geometry
Walker, Damien, and Lenk [85] argue that priors in Bayesian inference should be chosen to have Kullback-Leibler support for all θ ∈ Θ, that is they should have global Kullback-Leibler support. Moreover, the non-robustness mechanisms presented in Section 2.3 suggest the importance of selecting priors that are not "spread too thin". In this section we will discuss the relationship between these two notions under the condition that the model be measurable, injective, and open on its image in the weak topology. Barron, Schervish and Wasserman [3], Petrone and Wasserman [71], Ghosal, Ghosh and Ramamoorthi [36], and Wu and Ghosal [90], demonstrate that priors with global Kullback-Leibler support exist in many important cases. However, in general, the existence of such priors appears to be nontrivial. Indeed recall that, just before Corollary 3.1, it was established that the measurability of the model in the weak topology implies the measurability of the Kullback-Leibler neighborhoods K (θ) for θ ∈ Θ and > 0. However, the existence of strictly positive measures is also nontrivial and is connected with the Suslin Conjecture, see e.g. [54,32]. Evidently, the foundations for this subject were developed in Kelley [54] and have been well-developed in Comfort and Negrepontis [14]. According to Argyros [2], the first example of a compact space satisfying the countable chain condition, i.e. such that every pairwise disjoint collection of non-empty open subsets is countable, and not carrying a strictly positive measure, was given by Gaifman [32]. Argyros [2] provides more examples under different conditions. Moreover, although it is well known that every compact topological group supports a strictly positive measure, Todorčevič [82] shows that the free topological group of the one-point compactification of the discrete space of size continuum does not support a strictly positive measure. On the other hand, Comfort and Negrepontis [14], van Casteren [83] and Plebanek [72], provide necessary and sufficient conditions for their existence. Finally, when Θ is a perfect (i.e. with no isolated points) compact metric space, Hebert and Lacey [46,Cor. 2.8] demonstrate that it possesses a continuous (i.e. vanishes on singletons) strictly positive measure. Therefore, it is relevant to note that we have proven the the following lemma. In the situation of Lemma 6.1, although Theorem 5.4 does not apply, an almost everywhere version of it can be proved under the the weaker assumption that π has almost global Kullback-Leibler support, that is π ∈ K ae as defined in (3.3).
The following result illustrates density properties of the sets of measures with Kullback-Leibler support.
Lemma 6.2. Let Θ and X be Borel subsets of Polish metric spaces and consider a measurable model P : Θ → M(X), where M(X) is equipped with weak topology, and the resulting Kullback-Leibler divergence K on Θ × Θ. Then the set of probability measures K(θ) with Kullback-Leibler support at θ is a convex dense subset of M(Θ) in the total variation topology. Moreover, if the set of probability measures K := ∩ θ∈Θ K(θ) with Kullback-Leibler support at all θ ∈ Θ is non-empty, it is a convex dense subset.

Appendix: Basic Prokhorov Geometry
We establish a basic mechanism to bound from below the Prokhorov distance between two measures based on the values of the measures on the neighborhood of a single set. Then, for any µ ∈ M(Z), we have Proof. If d P r (µ 1 , µ 2 ) ≥ α the assertion is proved, so let us assume that d P r (µ 1 , µ 2 ) < α. Then, denoting d * := d P r (µ 1 , µ 2 ), it follows from the assumption that µ(A d * ) ≤ δ, so that proving the assertion.
Lemma 7.2. Let S be a separable metric space. Then, for an S-valued random variable X we have α(X, s) = d P r (L(X), δ s ) where α is the Ky Fan metric and s denotes the random variable with constant value s.
. To obtain equality, suppose that d P r (δ x 1 , δ x 2 ) > d(x 1 , x 2 ). Then, for any d which satisfies d P r (δ x 1 , δ x 2 ) > d > d(x 1 , x 2 ) there exists a measurable set B such that Consequently, x 1 ∈ B, but d > d(x 1 , x 2 ) implies that x 2 ∈ B d , which implies the contradiction 1 > 1 + d .
. Moreover, V * is an open neighborhood of P θ * . Since X is a separable metric space, it follows that d P r metrizes the weak topology, and since V * is open, it is well known (see e.g. [3,86,35]) that there exists a uniformly consistent test of P θ * against V c * , see Schwartz [78] for the definition of uniformly consistent test. It follows trivially that there exists a uniformly consistent test of P θ * against V c * ∩ P (Θ).
Moreover, since P is injective it follows that O c = P −1 (V c * ). Therefore, there exists a uniformly consistent test of P θ * against V c * ∩ P (Θ) = {P θ : θ ∈ O c }. Since V * is open, it also follows that there is a Prokhorov metric ball B s (P θ * ) of radius s > 0 about P θ * such that B s (P θ * ) ⊂ V * . Now consider the Kullback-Leibler ball K τ (P θ * ) for τ < s 2 2 . It follows from Csiszar, Kemperman and Kullback's [16] improvement K ≥ 1 2 d 2 tv of Pinsker's inequality and the inequality d tv ≥ d P r , that K τ (P θ * ) ⊂ B s (P θ * ). Since then K τ (P θ * ) ⊂ B s (P θ * ) ⊂ V * it follows that Consider now the Kullback-Leibler neighborhood W τ (θ * ) ⊂ Θ of θ * defined by pulling K τ (P θ * ) back to Θ by the model P : Then the previous inequality states that Since the Kullback-Leibler neighborhoods are measurable in the weak topology and P is assumed measurable, it follows that W τ (θ * ) is measurable.
Therefore, O and W τ (θ * ) satisfy the assumptions of the sets V and and W in [78, Thm. 6.1]. Consequently, since by assumption, the prior π has Kullback-Leibler support, it follows that we can apply Schwartz' theorem [78, Thm. 6.1] to obtain the assertion for O and since U ⊃ O is measurable the assertion follows.
Since Θ is a separable metric space it follows that M(Θ) equipped with the Prokhorov metric is a separable metric space. Since a.s. convergence implies convergence in probability for random variables with values in a separable metric space, it follows that Since M(Θ) is a separable metric space it follows that M 2 (Θ) equipped with the Prokhorov metric is also a separable metric space. Therefore, since on separable metric spaces convergence in probability to a constant valued random variable is equivalent to the weak convergence of the corresponding set of laws to the Dirac mass situated at that value, see e.g. Dudley [24,Prop. 11.1.3], it follows that the convergence in probability, π n → δ θ * in P ∞ θ * − probability, is equivalent to the corresponding convergence of laws Finally, since the Proprokhorov metric d P rr on M 2 (Θ) metrizes the weak topology on M 2 (Θ) = M(M(Θ)), it follows that the latter is equivalent to
Consequently, from the triangle inequality (8.14) and the convergence (8.13), we conclude, for any´ > 0, that for n large enough we have Consequently, if this Bayesian inference is Proprokhorov robust, then for > 0, it follows from (8.19) and (8.10) that δ < 1 N 2 . The requirement that perturbations be admissible, that is determine members in Z ρ (π), implies that δ < ρ.