Distributed Variational Representation Learning
  The problem of distributed representation learning is one in which multiple
sources of information $X_1,\ldots,X_K$ are processed separately so as to learn
as much information as possible about some ground truth $Y$. We investigate
this problem from information-theoretic grounds, through a generalization of
Tishby's centralized Information Bottleneck (IB) method to the distributed
setting. Specifically, $K$ encoders, $K \geq 2$, compress their observations
$X_1,\ldots,X_K$ separately in a manner such that, collectively, the produced
representations preserve as much information as possible about $Y$. We study
both discrete memoryless (DM) and memoryless vector Gaussian data models. For
the discrete model, we establish a single-letter characterization of the
optimal tradeoff between complexity (or rate) and relevance (or information)
for a class of memoryless sources (the observations $X_1,\ldots,X_K$ being
conditionally independent given $Y$). For the vector Gaussian model, we provide
an explicit characterization of the optimal complexity-relevance tradeoff.
Furthermore, we develop a variational bound on the complexity-relevance
tradeoff which generalizes the evidence lower bound (ELBO) to the distributed
setting. We also provide two algorithms that allow to compute this bound: i) a
Blahut-Arimoto type iterative algorithm which enables to compute optimal
complexity-relevance encoding mappings by iterating over a set of
self-consistent equations, and ii) a variational inference type algorithm in
which the encoding mappings are parametrized by neural networks and the bound
approximated by Markov sampling and optimized with stochastic gradient descent.
Numerical results on synthetic and real datasets are provided to support the
efficiency of the approaches and algorithms developed in this paper.
