Maximum-and-Concatenation Networks
  While successful in many fields, deep neural networks (DNNs) still suffer
from some open problems such as bad local minima and unsatisfactory
generalization performance. In this work, we propose a novel architecture
called Maximum-and-Concatenation Networks (MCN) to try eliminating bad local
minima and improving generalization ability as well. Remarkably, we prove that
MCN has a very nice property; that is, \emph{every local minimum of an
$(l+1)$-layer MCN can be better than, at least as good as, the global minima of
the network consisting of its first $l$ layers}. In other words, by increasing
the network depth, MCN can autonomously improve its local minima's goodness,
what is more, \emph{it is easy to plug MCN into an existing deep model to make
it also have this property}. Finally, under mild conditions, we show that MCN
can approximate certain continuous functions arbitrarily well with \emph{high
efficiency}; that is, the covering number of MCN is much smaller than most
existing DNNs such as deep ReLU. Based on this, we further provide a tight
generalization bound to guarantee the inference ability of MCN when dealing
with testing samples.
