Estimators of Entropy and Information via Inference in Probabilistic
  Models
  Estimating information-theoretic quantities such as entropy and mutual
information is central to many problems in statistics and machine learning, but
challenging in high dimensions. This paper presents estimators of entropy via
inference (EEVI), which deliver upper and lower bounds on many information
quantities for arbitrary variables in a probabilistic generative model. These
estimators use importance sampling with proposal distribution families that
include amortized variational inference and sequential Monte Carlo, which can
be tailored to the target model and used to squeeze true information values
with high accuracy. We present several theoretical properties of EEVI and
demonstrate scalability and efficacy on two problems from the medical domain:
(i) in an expert system for diagnosing liver disorders, we rank medical tests
according to how informative they are about latent diseases, given a pattern of
observed symptoms and patient attributes; and (ii) in a differential equation
model of carbohydrate metabolism, we find optimal times to take blood glucose
measurements that maximize information about a diabetic patient's insulin
sensitivity, given their meal and medication schedule.
