Information Bottleneck and its Applications in Deep Learning
  Information Theory (IT) has been used in Machine Learning (ML) from early
days of this field. In the last decade, advances in Deep Neural Networks (DNNs)
have led to surprising improvements in many applications of ML. The result has
been a paradigm shift in the community toward revisiting previous ideas and
applications in this new framework. Ideas from IT are no exception. One of the
ideas which is being revisited by many researchers in this new era, is
Information Bottleneck (IB); a formulation of information extraction based on
IT. The IB is promising in both analyzing and improving DNNs. The goal of this
survey is to review the IB concept and demonstrate its applications in deep
learning. The information theoretic nature of IB, makes it also a good
candidate in showing the more general concept of how IT can be used in ML. Two
important concepts are highlighted in this narrative on the subject, i) the
concise and universal view that IT provides on seemingly unrelated methods of
ML, demonstrated by explaining how IB relates to minimal sufficient statistics,
stochastic gradient descent, and variational auto-encoders, and ii) the common
technical mistakes and problems caused by applying ideas from IT, which is
discussed by a careful study of some recent methods suffering from them.
