Properties of Minimizing Entropy
  Compact data representations are one approach for improving generalization of
learned functions. We explicitly illustrate the relationship between entropy
and cardinality, both measures of compactness, including how gradient descent
on the former reduces the latter. Whereas entropy is distribution sensitive,
cardinality is not. We propose a third compactness measure that is a compromise
between the two: expected cardinality, or the expected number of unique states
in any finite number of draws, which is more meaningful than standard
cardinality as it discounts states with negligible probability mass. We show
that minimizing entropy also minimizes expected cardinality.
