{'fraction_non_alphanumeric': 0.034555288461538464, 'fraction_numerical': 0.012577266483516484, 'mean_word_length': 4.527164887307236, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
When are Bayesian model probabilities overconfident?

Bayesian model comparison is often based on the posterior distribution over the set of compared models. This distribution is often observed to concentrate on a single model even when other measures of model fit or forecasting ability indicate no strong preference. Furthermore, a moderate change in the data sample can easily shift the posterior model probabilities to concentrate on another model. We document overconfidence in two high-profile applications in economics and neuroscience. To shed more light on the sources of overconfidence we derive the sampling variance of the Bayes factor in univariate and multivariate linear regression. The results show that overconfidence is likely to happen when i) the compared models give very different approximations of the data-generating process, ii) the models are very flexible with large degrees of freedom that are not shared between the models, and iii) the models underestimate the true variability in the data.

B ayesian inference has gained widespread popularity in recent decades, largely propelled by advances in computing power and efficient simulation algorithms like Markov Chain Monte Carlo (1) and Sequential Monte Carlo (2). The Bayesian approach to model comparison is theoretically attractive and the standard in many fields in both the natural and the social sciences. Some examples are (3) in Linguistics, (4) in Economics, and (5) in Neuroscience. Hypothesis testing is a special case of model selection. Problems with classical hypothesis testing and the so called reproducibility crisis has directed attention to Bayesian model selection as an alternative, see for example (6) and (7).
A posterior distribution over a set of models makes it straightforward to select one of the models for further study, or to average inference across the models using Bayesian Model Averaging (BMA,8). Bayesian model comparison based on posterior model probabilities has many attractive properties: i) it allows the compared models to be non-nested, ii) it is consistent when the data generating process is among the compared models (the M-closed perspective in 9), iii) it will asymptotically concentrate the posterior probability mass on the model closest to the data generating process when all compared models are misspecified (M-open perspective in 9), and iv) it has direct connections to out-of-sample forecasting performance and cross-validation (9)(10)(11)(12)(13). However, Bayesian model probabilities sometimes behave puzzling in practise in that the posterior model distribution often concentrates entirely on one model, giving the impression of overwhelming support for that model. At the same time other forms of model comparison, e.g. predictive performance on a test set, do not show nearly the same degree of discrimination. This overconfidence is part of the folklore among expert Bayesians, but remains largely undocumented in the scientific literature, barring brief passages such as in (14) or (15) who state 'In practice we have observed a tendency of BMA to be over confident in weighting models-assigning weights that are too close to zero or one'. Moreover, theoretical work on overconfidence in Bayesian model probabilities is scarce. Two recent exceptions are (16) and (17). The asymptotic behavior of Bayesian model comparison when the compared models are equally misspecified is explored in (16), showing random walk like behavior of the log Bayes factor in large samples. The overconfidence of Bayesian posteriors and Bayesian model probabilities is also highlighted by (17) who use bagging of posteriors to make Bayesian inference more robust.
Our paper sheds light on the sources of overconfidence of Bayesian model probabilities by deriving the sampling variance of the Bayes factor in linear regression. We show that overconfidence is likely to be happen when i) the compared models give very different approximations of the data-generating process, ii) the models are very flexible, i.e. have large degrees of freedom, and that complexity is not shared between the models, and iii) the models are unable to replicate the variability in the data generating process. We also extend the results to multivariate regression.
The next section motivates our study by showing disturbingly clear signs of overconfidence in a high-profile applications in macroeconomics (4) and neuroscience (18). The rest of the paper studies the sources of overconfidence mathematically through the between-sample variance of the Bayes factor for Bayesian linear regression models. Proofs of the results are given in the Supplementary material.

Significance Statement
Bayesian model inference holds a central place in a wide range of scientific fields. However, Bayesian model probabilities often behave paradoxically in that they tend to concentrate almost entirely on a single model even when other measures of model fit do not indicate large differences between models. We document this overconfidence in high-profile applications in macroeconomics and neuroscience, and characterize the sources of overconfidence in a regression setting.

Background and motivation
Bayesian model probabilities. Consider comparing a set of K models, M = {M1, ..., MK }, for the observed data y = (y1, ..., yn) T , each depending on a vector of model parameters θ k . A common way of doing Bayesian model comparison is to use the posterior distribution over the model set M, and p(θ k |M k ) is the prior distribution for θ k . The Bayes factor for comparing model M k to model M l is

Micro-based general equilibrium models in macroeconomics.
To illustrate the effect of overconfidence in real-world applications, we first consider a class of Dynamic Stochastic General Equilibrium (19) models widely used in economics. DSGE models are the main models used for policy analysis and prediction at essentially every major monetary and fiscal institution in the world. Bayesian model probabilities is the standard tool for model comparison and selection among DSGE models (19). The seven-variable Smets-Wouters (4) model is the de facto starting point for most DSGE models used in practical work. The Smets-Wouters model is a probability model for seven macroeconomic time series using a complex microfunded model based on optimizing representative agents in the economy with rational expectations. The model dynamics are driven by seven underlying latent time series shocks, such as shocks in technology and preferences. In (4), the base version of the model is compared with eight variants that restricts certain model parameters to known values. The posterior model probabilities based on the marginal likelihoods from (4) is given in Table 1, showing conclusive evidence in favor of model M3. To investigate if overconfidence is a concern we approximate the sampling distribution of the posterior model probabilities using the circular block boostrap for time series (20). Figure 1 shows results from 1000 bootstrap replicates. The vertical bars correspond to models and the horizontal stripes to bootstrap replicates. The bootstrap replicates have been sorted with respect to the probabilities of the baseline model. The colors represent the posterior model probabilities. For example, a row where one of the columns has a stripe of dark purple implies that the model in that column has strong support (Pr(M k |Data) > 0.9), for the given bootstrap replicate, and all other models weak support (Pr(M k |Data) ≤ 0.1). Figure 1 shows that the conclusion from Table 1, where model M3 came out as the sure winner, is far from robust. In a large fraction of  the bootstrap samples, we actually have Pr(M7|Data) > 0.9; there is also the same level of support for the base model in a non-negligible fraction of bootstrap replicates. Figure 2 displays the percentage of bootstrap replicates where there is strong support for one of the models (the posterior model probability, PMP, is larger than 0.9, 0.95 and 0.99, respectively), or where the evidence is inconclusive. If we take the outcome that one of the PMPs are larger than 0.99 as conclusive, then Figure 2 shows that we have conclusive evidence in 35% of the replicates, but spread over 5 different models. We therefore conclude that the model comparison in (4) suffers from overconfidence with misleadingly conclusive support for M3. The authors of (4) seem in fact unimpressed by the strong support for model M3 as they silently continue the remainder of the article with the analysis of the base model.

Causal brain interactions in neuroscience.
Dynamic Causal Models (DCM) (21) is a popular class of models that use data from functional MRI (fMRI) brain scans to investigate how brain regions interact during an experimental task. Bayesian posterior model probabilities are the recommended method of model comparison for DCMs (22)[Ch.37]. DCMs are used in (18) to analyze how three brain regions that are known to be Fig. 3. Two of the compared DCM models for the three brain auditory brain regions: i) left posterior temporal sulcus (P), ii) left anterior superior temporal sulcus (A) and iii) pars orbitalis of the inferior frontal gyru (F). Dashed orange arrows are endogenous task-unrelated connections which are present in all models, while solid black arrows are connections whose strength is modulated by the hearing task. The location of the auditory input is indicated by the red square with an arrow. associated with speech processing interact when hearing intelligible speech. We reanalyze their data, but exclude subject 5 since it is a duplicate of subject 4. This leaves 25 subjects for the analysis. Figure 3 displays two of the compared models. A particular question of interest is in which of the three regions the auditory input is localized, regardless of the presence or absence of connectivity patterns. Such posterior probabilities are obtained by summing over all possible connections for a given source location, and is presented in Table 2, which is identical to Table 1 in (18) even with subject 5 removed. According to Table 2 we are supposed to be absolutely certain that input only enters through region P. Figure 4 however shows that in a non-negligible fraction of bootstrap samples we actually obtain conclusive evidence for region A. Figure shows the bootstrapped sampling distribution for the log Bayes factor comparing the hypothesis P against A. The regions of evidence from the well known conservative Kass-Raftery scale (23) are also indicated in the figure. The sampling variance is very large, and the regions of weak or inconclusive support is but a small interval between large masses of very strong support for either of the two hypotheses. Most of the mass for strong evidence happens to be located on P in this example, but it is clear that the Bayes factor can very easily be overconfident.

Overconfidence in Gaussian linear regression models
Univariate response. We explore the sources of overconfidence of Bayesian model comparison by deriving the sampling variance of the Bayes factor for two compared models in a finite sample setting. To get tractable and easily interpretable results we consider Gaussian linear regression models with known error variances σ 2 1 and σ 2 2 ; Remark 2 discusses the case with unknown variances. The data generating process M * is given by where X * is a n × p * matrix of full rank. We compare the misspecified models

M1
: M2 : where Xi is known and of full rank pi, and βi is a pi × 1 vector of unknown regression coefficients. We use Zellner's g-prior for both models for some shrinkage constant g > 0, but it is straightforward to extend the results to general normal priors. The posterior is of the form Our first result is derived under the assumption that the error variances in the two models are known and equal. This case gives a particularly simple expression with interesting interpretation. See the Appendix for results when the model variances differ. Let x 2 = (x T x) 1/2 be the Euclidean norm of the vector x and A F = tr(A T A) the Frobenius norm of the matrix A. Theorem 1. The sampling mean and variance of the log Bayes factor for the two regression models in [2], assuming equal and fixed variance σ 2 , with respect to the data-generating process in [1] is Theorem 1 shows that the Bayes factor favors models that are KL-close to the data-generating process, which is in line with the general asymptotic result in for example (25,26), but also that it penalizes complex models.
More interestingly, Theorem 1 shows that the variance increases with: i) μ1 −μ2 2 2 /σ 2 , ii) H1 − H2 F , and iii) the variance ratio σ 2 * /σ 2 . We discuss each of these parts in turn. It is straightforward to prove thatμi minimizes the KL divergence of p(y|Mi) from p(y|M * ) (see the Appendix); hence, µi is the best approximation of p (y|M * ) that model Mi is capable of. It is also easy to show (see the Appendix) that KL(M1(μ1) M2(μ2)) = μ1 −μ2 2 2 /2σ 2 , the KL divergence between the best approximating models N(y|μ1, σ 2 ) and N(y|μ2, σ 2 ), which happens to be symmetric when σ1 = σ2. The first term in Theorem 1 therefore shows that the variance of the Bayes factor tends to be large when the two models approximate M * in widely different ways. This explains why continuous model expansion, where a model is embedded in a larger family via a continuous parameter, is preferred over a comparison of a discrete set of well separated models (27). A model is always surrounded by other similar models in continuous model expansions. We also note that the recommended strategy in (27) is to compare widely different models that 'stake out the corners in the model space' in order to capture the true model uncertainty. This is a good strategy when one can afford to stake out the corners with a dense set of models, preferably even a continuum of models, but this is rarely the case. A much more common situation is when the model space is staked out using a small set of models. Unfortunately, Theorem 1 shows that the posterior model probabilities are then highly likely to be overconfident.
To interpret the term H1 − H2 F recall that the degrees of freedom of a linear smoother,ŷ = Hy, is given by tr(H) (28). The next lemma shows that H1 − H2 2 F is the total degrees of freedom of the two models that is not shared between them.
By Lemma 1, H1 − H2 2 F is the total complexity of the two models, tr(P1)+tr(P2), reduced by the s completely shared dimensions and by the cosine of the principal angles of the r partially shared dimensions, shrunk by the precision of the prior. Hence, Var(log B12(y)) is not affected by any complexity that is shared between the models. Comparison of a dense set of nested models, e.g. variable selection in regression, is therefore expected to less prone to overconfidence since there is often a large overlap between compared models. Figure 6 gives an abstract illustration of models, divergences between models and their shared/non-shared complexities.
Finally, both μ1 −μ2 2 2 /σ 2 and H1 − H2 F in Theorem 1 are inflated by the error variance ratio σ 2 * /σ 2 in the expression for Var(log B12(y)). Models that are unable to generate enough variation in their data distribution are therefore particularly susceptible to overconfidence.
We make the following additional remarks about Theorem 1.

Remark 1.
Var(log B12(y)) does not directly depend on the degree of misspecification of the two models relative to the true M * ; only the divergence between the models matters. However, the degree of misspecification restricts how far apart the models can be. One way to see this is by noting that μ1 −μ2 2 2 has the upper bound which under equal degree of misspecification simplifies to Hence, in problems where all models are very misspecified there is greater scope for the models to approximate p(y|M * ) in markedly different ways, and therefore greater risk of overconfidence.

Remark 2.
Since the known variance σ 2 enters the mean and variance in Theorem 1 as a multiplicative factor 1/σ 2 , it is straightforward to generalize Theorem 1 to the case with a common and unknown variance by applying the law of total variance. The end result is that the factor 1/σ 2 is replaced by its prior expectation, but the two main terms μ1 −μ2 2 2 /σ 2 and H1 − H2 F remain. The case with unknown and different variances seems to be intractable, but numerical experiments indicate that the same three factors are driving Var(log B12(y)).

Remark 3. The technique behind Theorem 1 can also be used for regression models with heteroscedastic variance as the log of the marginal likelihood remains quadratic in y with
where Σε is the n × n covariance matrix of errors. This framework includes the popular Gaussian process regression models in machine learning where and the function f (x) follows a Gaussian process with a given covariance kernel. The log marginal likelihood is given in Equation 2.43 of (29).  Hi). The red points representμi, the best approximation to M * within Mi. Finally, the dotted line illustrates the Kullback-Leibler divergence between the models, i.e. ||μ1 −μ2||. The left graph shows models with no shared complexity, the middle graph shows models with substantial shared complexity, and the graph to the right shows models without shared complexity where one of the models is closer to M * .
Multivariate response. The previous section shows that overconfidence is a bigger concern when the compared models are approximating the data-generating process in very different ways. The DSGE models from (4) are seemingly similar, however; they only differ from the baseline model by setting a single parameter to a specific value. Note however that the results in the previous section are derived for the case with a univariate response, while the DSGE models are multivariate model with seven time series responses. This subsection extends the previous results to multivariate regression and highlights some special properties for this more general case. The data-generating process M * is given by the multivariate regression and we consider comparing the following models where N(µ, A1, A2) denotes the matrix variate distribution, Y is n×q , Xi is known and of full rank pi, and Bi is a pi×q vector of unknown parameters. The error term Ei = (ε1i, ε2i, ..., εni) is an n × q matrix, following a matrix normal distribution, with rows that are iid N(0, Σi). We let βi = vec(Bi), and use Zellner's g-prior for the regression coefficients.

Theorem 2. The sampling mean and variance of the log
Bayes factor for the two multivariate regression models in [4] with respect to the data-generating process in [3] is where Ω ≡ Σ −1/2 Σ * Σ −1/2 is a multivariate generalization of the variance ratio σ 2 * /σ 2 andμi = Hiµ * is the projection of the true mean vector µ * onto model Mi.
The interpretation remains largely the same as in the univariate case, with the added insight that not all differences in the models are equally important due to the appearance of the generalized variance ratio Ω in (μ2 −μ1)Σ −1/2 Ω 1/2 2 F . To show this more precisely, we perform the two spectral decompositions: where U = (u1, ..., up) and U * = (u * 1, ..., u * p) are matrices of eigenvectors, and Λ = Diag(λ1, ..., λp) and Λ * = Diag(λ * 1, ..., λ * p) are diagonal matrices of eigenvalues. Now, The n × p matrix (μ1 −μ2) U Λ −1/2 contains the differences in predictions in the directions of the principal components of Σ, rescaled to unit variance. The p×p matrix Λ −1/2 U T U * Λ 1/2 * has element λ * j/λi cos(φu i ,u * j ) in its ith row, jth column, where cos(φu i ,u * j ) = u T i u * j measures the degree of alignment of pairs of eigenvectors from Σ and Σ * . Hence, var log B12(Y ) will be large when the models make very different prediction on linear combinations of response variables where the eigenvectors of Σ and Σ * align, and the variance ratio λ * j/λi is large. This agrees with the analysis of forecasting performance in (30) for an open-economy extension of the Smets-Wouters model. In addition, (30) show that a multivariate measure of out-ofsample forecasting performance is almost entirely driven by forecasting errors in employment, one of the least important variables from a central bank perspective.

Discussion and Conclusion
We have demonstrated that Bayesian posterior model probabilities can be overconfident in the sense of spuriously picking out one of the compared models as the only probable model in a set of compared models, while at the same time being equally certain about another model in a slightly different dataset. We have analyzed the sources of this overconfidence by deriving the sample variance of the log Bayes factor for univariate and multivariate regression.
The main message is that overconfidence is likely to be a problem when the compared models give very different approximations of the data-generating process and when the compared models are flexible in a way that is not shared among the models. The same is true for the multivariate setting, with the added nuance that overconfidence will be largest when the models are different with respect to specific linear combinations of the response variables.
Our results motivate several interesting avenues for future research. First, the linear regression setup was chosen since it provides a clear view of what drives overconfidence. It would be illuminating to derive similar measures for more general models to see if the same effects appear there. Second, it would of interest to repeat our analysis for other Bayesian model inference methods, for example prediction pools or stacking (14,31). Stacking is particularly interesting since it is known to not necessarily concentrate on a single model when the sample grows large. Third, we have used the bootstrap to approximate the sampling distribution of Bayes factors. This can be very time-consuming when models are analyzed by Markov Chain Monte Carlo (MCMC), since we would have to run the MCMC for each bootstrap sample. It would therefore be of practical importance to explore the efficiency of methods where the marginal likelihood for each bootstrap sample is obtained by reweighting the posterior draws from a single MCMC run on the original dataset (32).
Proof of Theorem 1. The log Bayes factor is equal to where Qi(y) = 1 2σ 2 i y T (In − Hi)y and Hi = κPi with Pi = Xi X T i Xi −1 X T i being a symmetric and idempotent projection matrix. Using Part i of Lemma 2, we have E(log B12(y)) = n 2 log σ 2 A. Proof of Theorem 2. We derive the result for the case Σ1 = Σ2. The proof when Σ1 = Σ2 is similar, but more tedious algebraically. The log Bayes factor is where vec(Y ) ∼ N (vec(µ * ), Σ * ⊗ In), µ * = X * B * , and Hi = κPi with Pi = Xi X T i Xi −1 X T i . Using Lemma 2, we obtain the mean as Similarly to the proof of the univariate case in Theorem 1, we can relate the terms tr µ T * Hiµ * Σ −1 to the Kullback-Leibler divergence between p(Y |µ * ) and p(Y |μi). The Kullback-Leibler divergence of p(Y |μi) from p(Y |µ * ) can be derived by employing the usual KL divergence between multivariate normals in (34)  where Ω ≡ Σ −1/2 Σ * Σ −1/2 is a multivariate generalization of the variance ratio σ 2 * /σ 2 .