A type of generalization error induced by initialization in deep neural
  networks
  How initialization and loss function affect the learning of a deep neural
network (DNN), specifically its generalization error, is an important problem
in practice. In this work, by exploiting the linearity of DNN training dynamics
in the NTK regime \citep{jacot2018neural,lee2019wide}, we provide an explicit
and quantitative answer to this problem. Focusing on regression problem, we
prove that, in the NTK regime, for any loss in a general class of functions,
the DNN finds the same \emph{global} minima---the one that is nearest to the
initial value in the parameter space, or equivalently, the one that is closest
to the initial DNN output in the corresponding reproducing kernel Hilbert
space. Using these optimization problems, we quantify the impact of initial
output and prove that a random non-zero one increases the generalization error.
We further propose an antisymmetrical initialization (ASI) trick that
eliminates this type of error and accelerates the training. To understand
whether the above results hold in general, we also perform experiments for DNNs
in the non-NTK regime, which demonstrate the effectiveness of our theoretical
results and the ASI trick in a qualitative sense. Overall, our work serves as a
baseline for the further investigation of the impact of initialization and loss
function on the generalization of DNNs, which can potentially guide and improve
the training of DNNs in practice.
