Given the posterior predictive distribution for a new data point $x^*$, the posterior predictive distribtion given some data $(X,Y)$

\begin{align*} p(y^*|x^*,X,Y) = \int p(y^*|x^*,\omega) p(\omega|X,Y) \, d \omega \end{align*}

gives us the distribution of future predicted data $y^*$.

What is the logic behind the integral? Why do we need an integral here?


EDIT: What is the intuition behind the integral? Obviously it measures some kind of area within $p(y^*|x^*,\omega) p(\omega|X,Y)$, which are nothing more than mass functions.

  • 1
    $\begingroup$ Perhaps $\omega$ is unobserved so the best you can do is integrate it out using a distribution $p(\omega \lvert X,Y)$ conditioned on the variables you observe $X$,$Y$. But unless you provide more detail on the set up, the is simply guessing. $\endgroup$ – Jesper for President May 27 '20 at 8:14
  • $\begingroup$ @JesperforPresident thanks for your comment. The new data point $x^*$ depends on parameter $\omega$, whereas the posterior distribution of $\omega$ depends on given data $X$. The definition of the posterior predictive distribution I used is from wikipedia. Perhaps the intuition of using an integral here isn't clear to me. $\endgroup$ – MJimitater May 27 '20 at 8:21

You don't exactly know $\omega$ but you have some idea, a distribution based on the previous data you've seen, which is described by $p(\omega|X,Y)$. If you had a constant $\omega_0$, the posterior predictive distribution would be $p(y^*|x^*,\omega_0)$, but the integral is basically an expected value (i.e. a weighted average) over all possible $\omega$.

By the way, the integral is at the same time comes from total probability law: $$p(y^*|x^*,X,Y)=\int \underbrace{p(y^*|x^*,\omega,X,Y)p(\omega|X,Y)}_{p(y,\omega|x^*,X,Y)}d\omega=\int p(y^*|x^*,\omega)p(\omega|X,Y)d\omega$$

The first term inside the integral is simplified as $p(y^*|x^*,\omega,X,Y)=p(y^*|x^*,\omega)$, because when you actually know the model parameters, you don't need the training data to learn them. So, given the input $x^*$, the output $y^*$ is assumed to be dependent on only the model parameters, $\omega$.


Your Answer

