On the Computational Complexity of High-Dimensional Bayesian Variable
  Selection
  We study the computational complexity of Markov chain Monte Carlo (MCMC)
methods for high-dimensional Bayesian linear regression under sparsity
constraints. We first show that a Bayesian approach can achieve
variable-selection consistency under relatively mild conditions on the design
matrix. We then demonstrate that the statistical criterion of posterior
concentration need not imply the computational desideratum of rapid mixing of
the MCMC algorithm. By introducing a truncated sparsity prior for variable
selection, we provide a set of conditions that guarantee both
variable-selection consistency and rapid mixing of a particular
Metropolis-Hastings algorithm. The mixing time is linear in the number of
covariates up to a logarithmic factor. Our proof controls the spectral gap of
the Markov chain by constructing a canonical path ensemble that is inspired by
the steps taken by greedy algorithms for variable selection.
