TITLE: How to compute evidence lower bound (ELBO) when the complete log-likelihood is intractable?
QUESTION [1 upvotes]: As an example, assume that I have data $\mathbf{X}$, unobserved variables $\mathbf{Z}$ and model parameters $\pmb{\alpha}$, $\pmb{\beta}$. I am omitting the mentioning of any variational parameters. The variational distribution $q$ is over $\mathbf{Z}$, $\pmb{\alpha}$, and $\pmb{\beta}$. Also, let's say that the distribution over the latent variables contains an intractable normalization constant $\Omega(\pmb{\beta})$:
\begin{align}
p(\mathbf{Z} | \pmb{\beta}) = \frac{1}{\Omega(\pmb{\beta})}f(\mathbf{Z})\hat{p}(\mathbf{Z}|\pmb{\beta})
\end{align}
I now want to compute the ELBO:
\begin{align}
\text{ELBO}(q)
&= \text{E}_q[\log p(\mathbf{X}, \mathbf{Z}, \pmb{\alpha}, \pmb{\beta})] - \text{E}_q[\log q(\mathbf{Z}, \pmb{\alpha}, \pmb{\beta})] \\
&= \text{E}_q[\log p(\mathbf{X}| \mathbf{Z}, \pmb{\alpha})] + \color{red}{\text{E}_q[\log p(\mathbf{Z}| \pmb{\beta})]} + \text{E}_q[\log p(\pmb{\alpha})] + \text{E}_q[\log p(\pmb{\beta})] - \text{E}_q[\log q(\mathbf{Z}, \pmb{\alpha}, \pmb{\beta})]
\end{align}
My problem is with the term in red. I can try to decompose it as:
\begin{align}
\color{red}{\text{E}_q[\log p(\mathbf{Z}| \pmb{\beta})]} 
&= \text{E}_q[\log f(\mathbf{Z})] + \text{E}_q[\log \hat{p}(\mathbf{Z}| \pmb{\beta})] - \text{E}_q[\log \Omega(\pmb{\beta})]
\end{align}
But how can I deal with $\text{E}_q[\log \Omega(\pmb{\beta})]$? I am using the ELBO as a termination criterion for variational Bayes. Since the ELBO is a lower bound on the evidence and I can assume $\Omega(\pmb{\beta}) > 0 \text{,  }\forall \pmb{\beta}$, since it's a normalization constant, I am thinking of simply dropping this term. Any thoughts?

REPLY [0 votes]: So I think you may need to clarify some of your formulae.
If $\pmb\alpha$ and $\pmb\beta$ are parameters, then they shouldn't have probability distributions associated with them.
If $\pmb\beta$ is in fact a parameter, then you simply have,
$$\text{E}_q[\log\Omega(\pmb\beta)]=\log\Omega(\pmb\beta).$$
{"set_name": "stack_exchange", "score": 1, "question_id": 3819228}