Exercise - Variational Mean Field Approximation for Univariate Gaussian

Introduction

[TODO]

In order to detect errors in your own code, execute the notebook cells containing assert or assert_almost_equal.

Requirements

[TODO]

Python Modules

%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
import scipy.stats
np.random.seed(40)

Data

Data: $X \sim \mathcal N(\mu, \frac{1}{\tau})$

Probability Density Function: $p(X \mid \mu, \tau) = \sqrt{\frac{\tau}{2\pi}} \exp\left( -\frac{\tau (X-\mu)^2 }{2} \right)$

with
-$\mu$: mean -$\sigma^2$: variance -$\tau =\frac{1}{\sigma^2}$ : precision

# generation of observed data
N = 10
mu = 10.
sigma = 2.
X = np.random.normal(mu, sigma, N)
x = np.arange(3,18,0.01)
p_x = scipy.stats.norm.pdf(x, loc=mu, scale=sigma)
plt.plot(x, p_x, label="true Gaussian")
plt.plot(X, np.zeros_like(X), "ro", label="Data")
plt.title("")
plt.xlabel("x")
plt.ylabel("p(x)")
plt.legend();

Exercises

Exercise - Mean Field Approximation of the Posterior

Find the mean field approximation of the posterior:

$p(\mu, \tau \mid X) \approx q(\mu)q(\tau)$.

• The observed data was sampled from a Gaussian distribution:$X \sim \mathcal N(\mu, \frac{1}{\tau})$.
• Use a constant prior for the mean and the precision$\tau = \frac{1}{\sigma^2}$: $p(\mu, \tau) = const. \quad \text{ for } \tau > 0$ $\theta = (\theta_0, \theta_1) = (\mu, \tau)$

Note: Typically the mean-field approximation in closed form is done with the conjugate distributions. But here we use the constant (improper) prior because it's a little bit easier.

Recap: Mean field approximation

Loop until convergence: $\log q({\theta}_k) = \mathbb E_{q_{-k}} \left[ \log{\hat p( {\theta} \mid {\mathcal D} )} \right] + const.$

Hint:

Use the "Sum of difference of the mean": $\sum_{i=1}^n (x_i-\mu)^2 = \sum_{i=1}^n(x_i-\bar{x})^2 + n(\bar{x} -\mu)^2$ with $\bar{x} = \frac{1}{n}\sum_{i=1}^n x_i$

Exercise - Proof

Show that: $q(\mu) = \mathcal N(\bar{X}, \frac{1}{\gamma_1})$ with$\gamma_1 = m \mathbb E_{q_{\tau}}[\tau]$

and

$q(\tau) = \text{Gamma}(\frac{m}{2}+1, \frac{2}{\gamma_2})$

with $\gamma_2= \sum_{i=1}^m \left(X_i^2\right)-m\bar X^2 + m\text{var}_{q_{\mu}}(\mu)$

Gamma distribution

$\text{Gamma}(k, \theta') = p(x) = x^{k-1}\frac{e^{-x/\theta'}}{\theta'^k\Gamma(k)}$ -$k$ is the shape -$\theta'$ the scale, -$\Gamma(.)$ is the Gamma function

Exercise - Implementation of the Mean Field Approximation

Implement the mean field approximation. Like always, you are free to implement as many helper functions as you want.

If everything is correct, executing the cells below should plot figures similar to these:

def optim(X=X,  mean_mu=1, sigma_quare_mu=1, loc_tau=1.,  scale_tau=1.):
raise NotImplementedError()
mean_mu, sigma_quare_mu, loc_tau, scale_tau = optim(X)
mean_mu, sigma_quare_mu, loc_tau, scale_tau
plt.figure(figsize=(12,4))

x = np.arange(5,15,0.01)
p_mu = scipy.stats.norm.pdf(x, loc=mean_mu, scale=np.sqrt(sigma_quare_mu))
ax = plt.subplot(121)
ax.plot(x, p_mu)
ax.set_xlabel("$\mu$")
ax.set_ylabel("q($\mu$)")
ax.set_title("Mean: q($\\mu$)")
print("true mu: ", mu)

x = np.arange(0,1,0.01)
p_tau = scipy.stats.gamma.pdf(x, a=loc_tau, scale=scale_tau)
ax = plt.subplot(122)
ax.plot(x, p_tau)
ax.set_xlabel("$\\tau$")
ax.set_ylabel("q($\\tau$)")
ax.set_title("Precision: q($\\tau$)")
print("true tau: ", 1/sigma**2)

Literature

[TODO]

The following license applies to the complete notebook, including code cells. It does however not apply to any referenced external media (e.g., images).

Exercise - Variational Mean Field Approximation for Univariate Gaussian
by Christian Herta
The text passage is a detailed description of an exercise on variational mean field approximation for a univariate Gaussian distribution, including mathematical derivations, implementation instructions, and code snippets for plotting results.
no_question

the passage does not contain a direct mathematical question or a request for a specific calculation. instead, it provides a set of exercises and instructions for implementing a mean field approximation, but does not ask a question that requires a specific answer within the text.
Given a text passage, determine:
    1. If it contains a mathematical question (including word problems or calculations)
    2. If an answer to that mathematical question is provided in the text

First, describe the content structure you found in 1-2 sentences.

Then provide your analysis in the following format:
TYPE: <one of the following values>
- NO_QUESTION
- SINGLE_QUESTION
- SINGLE_QUESTION_WITH_ANSWER
- MULTIPLE_QUESTIONS
- MULTIPLE_QUESTIONS_WITH_ANSWERS
system
Exercise - Variational Mean Field Approximation for Univariate Gaussian

Introduction

[TODO]

In order to detect errors in your own code, execute the notebook cells containing assert or assert_almost_equal.

Requirements

[TODO]

Python Modules

%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
import scipy.stats
np.random.seed(40)

Data

Data: $X \sim \mathcal N(\mu, \frac{1}{\tau})$

Probability Density Function: $p(X \mid \mu, \tau) = \sqrt{\frac{\tau}{2\pi}} \exp\left( -\frac{\tau (X-\mu)^2 }{2} \right)$

with
-$\mu$: mean -$\sigma^2$: variance -$\tau =\frac{1}{\sigma^2}$ : precision

# generation of observed data
N = 10
mu = 10.
sigma = 2.
X = np.random.normal(mu, sigma, N)
x = np.arange(3,18,0.01)
p_x = scipy.stats.norm.pdf(x, loc=mu, scale=sigma)
plt.plot(x, p_x, label="true Gaussian")
plt.plot(X, np.zeros_like(X), "ro", label="Data")
plt.title("")
plt.xlabel("x")
plt.ylabel("p(x)")
plt.legend();

Exercises

Exercise - Mean Field Approximation of the Posterior

Find the mean field approximation of the posterior:

$p(\mu, \tau \mid X) \approx q(\mu)q(\tau)$.

• The observed data was sampled from a Gaussian distribution:$X \sim \mathcal N(\mu, \frac{1}{\tau})$.
• Use a constant prior for the mean and the precision$\tau = \frac{1}{\sigma^2}$: $p(\mu, \tau) = const. \quad \text{ for } \tau > 0$ $\theta = (\theta_0, \theta_1) = (\mu, \tau)$

Note: Typically the mean-field approximation in closed form is done with the conjugate distributions. But here we use the constant (improper) prior because it's a little bit easier.

Recap: Mean field approximation

Loop until convergence: $\log q({\theta}_k) = \mathbb E_{q_{-k}} \left[ \log{\hat p( {\theta} \mid {\mathcal D} )} \right] + const.$

Hint:

Use the "Sum of difference of the mean": $\sum_{i=1}^n (x_i-\mu)^2 = \sum_{i=1}^n(x_i-\bar{x})^2 + n(\bar{x} -\mu)^2$ with $\bar{x} = \frac{1}{n}\sum_{i=1}^n x_i$

Exercise - Proof

Show that: $q(\mu) = \mathcal N(\bar{X}, \frac{1}{\gamma_1})$ with$\gamma_1 = m \mathbb E_{q_{\tau}}[\tau]$

and

$q(\tau) = \text{Gamma}(\frac{m}{2}+1, \frac{2}{\gamma_2})$

with $\gamma_2= \sum_{i=1}^m \left(X_i^2\right)-m\bar X^2 + m\text{var}_{q_{\mu}}(\mu)$

Gamma distribution

$\text{Gamma}(k, \theta') = p(x) = x^{k-1}\frac{e^{-x/\theta'}}{\theta'^k\Gamma(k)}$ -$k$ is the shape -$\theta'$ the scale, -$\Gamma(.)$ is the Gamma function

Exercise - Implementation of the Mean Field Approximation

Implement the mean field approximation. Like always, you are free to implement as many helper functions as you want.

If everything is correct, executing the cells below should plot figures similar to these:

def optim(X=X,  mean_mu=1, sigma_quare_mu=1, loc_tau=1.,  scale_tau=1.):
raise NotImplementedError()
mean_mu, sigma_quare_mu, loc_tau, scale_tau = optim(X)
mean_mu, sigma_quare_mu, loc_tau, scale_tau
plt.figure(figsize=(12,4))

x = np.arange(5,15,0.01)
p_mu = scipy.stats.norm.pdf(x, loc=mean_mu, scale=np.sqrt(sigma_quare_mu))
ax = plt.subplot(121)
ax.plot(x, p_mu)
ax.set_xlabel("$\mu$")
ax.set_ylabel("q($\mu$)")
ax.set_title("Mean: q($\\mu$)")
print("true mu: ", mu)

x = np.arange(0,1,0.01)
p_tau = scipy.stats.gamma.pdf(x, a=loc_tau, scale=scale_tau)
ax = plt.subplot(122)
ax.plot(x, p_tau)
ax.set_xlabel("$\\tau$")
ax.set_ylabel("q($\\tau$)")
ax.set_title("Precision: q($\\tau$)")
print("true tau: ", 1/sigma**2)

Literature

[TODO]

The following license applies to the complete notebook, including code cells. It does however not apply to any referenced external media (e.g., images).

Exercise - Variational Mean Field Approximation for Univariate Gaussian
by Christian Herta
user
The text passage is a detailed description of an exercise on variational mean field approximation for a univariate Gaussian distribution, including mathematical derivations, implementation instructions, and code snippets for plotting results.

TYPE: NO_QUESTION

The passage does not contain a direct mathematical question or a request for a specific calculation. Instead, it provides a set of exercises and instructions for implementing a mean field approximation, but does not ask a question that requires a specific answer within the text.