Drill the Cork of Information Bottleneck by Inputting the Most Important
  Data
  Deep learning has become the most powerful machine learning tool in the last
decade. However, how to efficiently train deep neural networks remains to be
thoroughly solved. The widely used minibatch stochastic gradient descent (SGD)
still needs to be accelerated. As a promising tool to better understand the
learning dynamic of minibatch SGD, the information bottleneck (IB) theory
claims that the optimization process consists of an initial fitting phase and
the following compression phase. Based on this principle, we further study
typicality sampling, an efficient data selection method, and propose a new
explanation of how it helps accelerate the training process of the deep
networks. We show that the fitting phase depicted in the IB theory will be
boosted with a high signal-to-noise ratio of gradient approximation if the
typicality sampling is appropriately adopted. Furthermore, this finding also
implies that the prior information of the training set is critical to the
optimization process and the better use of the most important data can help the
information flow through the bottleneck faster. Both theoretical analysis and
experimental results on synthetic and real-world datasets demonstrate our
conclusions.
