Neural Capacity Estimators: How Reliable Are They?
  Recently, several methods have been proposed for estimating the mutual
information from sample data using deep neural networks and without the knowing
closed form distribution of the data. This class of estimators is referred to
as neural mutual information estimators. Although very promising, such
techniques have yet to be rigorously bench-marked so as to establish their
efficacy, ease of implementation, and stability for capacity estimation which
is joint maximization frame-work. In this paper, we compare the different
techniques proposed in the literature for estimating capacity and provide a
practitioner perspective on their effectiveness. In particular, we study the
performance of mutual information neural estimator (MINE), smoothed mutual
information lower-bound estimator (SMILE), and directed information neural
estimator (DINE) and provide insights on InfoNCE. We evaluated these algorithms
in terms of their ability to learn the input distributions that are capacity
approaching for the AWGN channel, the optical intensity channel, and peak
power-constrained AWGN channel. For both scenarios, we provide insightful
comments on various aspects of the training process, such as stability,
sensitivity to initialization.
