An Equivalence between Bayesian Priors and Penalties in Variational
  Inference
  In machine learning, it is common to optimize the parameters of a
probabilistic model, modulated by an ad hoc regularization term that penalizes
some values of the parameters. Regularization terms appear naturally in
Variational Inference (VI), a tractable way to approximate Bayesian posteriors:
the loss to optimize contains a Kullback--Leibler divergence term between the
approximate posterior and a Bayesian prior. We fully characterize which
regularizers can arise this way, and provide a systematic way to compute the
corresponding prior. This viewpoint also provides a prediction for useful
values of the regularization factor in neural networks. We apply this framework
to regularizers such as L2, L1 or group-Lasso.
