Confidence Dimension for Deep Learning based on Hoeffding Inequality and
  Relative Evaluation
  Research on the generalization ability of deep neural networks (DNNs) has
recently attracted a great deal of attention. However, due to their complex
architectures and large numbers of parameters, measuring the generalization
ability of specific DNN models remains an open challenge. In this paper, we
propose to use multiple factors to measure and rank the relative generalization
of DNNs based on a new concept of confidence dimension (CD). Furthermore, we
provide a feasible framework in our CD to theoretically calculate the upper
bound of generalization based on the conventional Vapnik-Chervonenk dimension
(VC-dimension) and Hoeffding's inequality. Experimental results on image
classification and object detection demonstrate that our CD can reflect the
relative generalization ability for different DNNs. In addition to
full-precision DNNs, we also analyze the generalization ability of binary
neural networks (BNNs), whose generalization ability remains an unsolved
problem. Our CD yields a consistent and reliable measure and ranking for both
full-precision DNNs and BNNs on all the tasks.
