Dissecting Non-Vacuous Generalization Bounds based on the Mean-Field
  Approximation
  Explaining how overparametrized neural networks simultaneously achieve low
risk and zero empirical risk on benchmark datasets is an open problem.
PAC-Bayes bounds optimized using variational inference (VI) have been recently
proposed as a promising direction in obtaining non-vacuous bounds. We show
empirically that this approach gives negligible gains when modeling the
posterior as a Gaussian with diagonal covariance--known as the mean-field
approximation. We investigate common explanations, such as the failure of VI
due to problems in optimization or choosing a suboptimal prior. Our results
suggest that investigating richer posteriors is the most promising direction
forward.
