Feature-Dependent Cross-Connections in Multi-Path Neural Networks
  Learning a particular task from a dataset, samples in which originate from
diverse contexts, is challenging, and usually addressed by deepening or
widening standard neural networks. As opposed to conventional network widening,
multi-path architectures restrict the quadratic increment of complexity to a
linear scale. However, existing multi-column/path networks or model ensembling
methods do not consider any feature-dependent allocation of parallel resources,
and therefore, tend to learn redundant features. Given a layer in a multi-path
network, if we restrict each path to learn a context-specific set of features
and introduce a mechanism to intelligently allocate incoming feature maps to
such paths, each path can specialize in a certain context, reducing the
redundancy and improving the quality of extracted features. This eventually
leads to better-optimized usage of parallel resources. To do this, we propose
inserting feature-dependent cross-connections between parallel sets of feature
maps in successive layers. The weighting coefficients of these
cross-connections are computed from the input features of the particular layer.
Our multi-path networks show improved image recognition accuracy at a similar
complexity compared to conventional and state-of-the-art methods for deepening,
widening and adaptive feature extracting, in both small and large scale
datasets.
