We can express the KL divergence between q$_{GLYPH<21>}$ and p as where we define Here, ❊ q$_{GLYPH<21>}$ denotes an average over GLYPH<12>$_{1:}$$_{H}$ , using q .GLYPH<12>$_{1:}$$_{H}$ j GLYPH<21>/ . Because KL $^{[}$q$_{GLYPH<21>}$ jj p ] is non- negative, (10) implies that, for all distributions q$_{GLYPH<21>}$ , The likelihood is called the “evidence” in some contexts; we therefore christen L .GLYPH<21> I GLYPH<16> ; GLYPH<127>/ the evidence lower bound (ELBO) function. Using (10), we can formulate a maximization problem equivalent to (8), having the same optimal point q .GLYPH<12>$_{1:}$$_{H}$ j GLYPH<21> $^{GLYPH<3>}$/ : The equivalence of (8) and (13), together with the bound (12), shows that the best approxi- mation in Q to the posterior yields the tightest lower bound on the marginal likelihood. We work with (13) rather than (8) – to evaluate the KL divergence, we would need to compute the marginal likelihood, bringing us back to our original problem. The variational parameters to be adjusted are GLYPH<21> D f GLYPH<22>$_{1:}$$_{H}$ ; 6$_{1:}$$_{H}$ g . We conduct the variational inference (13) using block coordinate ascent on the coordinate blocks GLYPH<22>$_{1}$ , 6$_{1}$ , . . . , GLYPH<22>$_{H}$ , 6$_{H}$ . The coor- dinate updates do not have a closed form, but each update solves a smooth, unconstrained 8 KL $^{[}$q$_{GLYPH<21>}$ jj p ] D ❊ q$_{GLYPH<21>}$ log$^{[}$ q .GLYPH<12>$_{1:}$$_{H}$ j GLYPH<21>/ p .GLYPH<12>$_{1:}$$_{H}$ j D ; GLYPH<16> ; GLYPH<127>/$^{]}$ (9) D GLYPH<0> L .GLYPH<21> I GLYPH<16> ; GLYPH<127>/ C log p . D j GLYPH<16> ; GLYPH<127>/ (10) L .GLYPH<21> I GLYPH<16> ; GLYPH<127>/ : D GLYPH<0> ❊ q$_{GLYPH<21>}$ log$^{[}$ q .GLYPH<12>$_{1:}$$_{H}$ j GLYPH<21>/ p .GLYPH<12>$_{1:}$$_{H}$ ; D j GLYPH<16> ; GLYPH<127>/$^{]}$ : (11) L .GLYPH<21> I GLYPH<16> ; GLYPH<127>/ GLYPH<20> log p . D j GLYPH<16> ; GLYPH<127>/ : (12) GLYPH<21> GLYPH<3> D argmax GLYPH<21> 2 3 L .GLYPH<21> I GLYPH<16> ; GLYPH<127>/ : (13)