Greedy Shallow Networks: An Approach for Constructing and Training
  Neural Networks
  We present a greedy-based approach to construct an efficient single hidden
layer neural network with the ReLU activation that approximates a target
function. In our approach we obtain a shallow network by utilizing a greedy
algorithm with the prescribed dictionary provided by the available training
data and a set of possible inner weights. To facilitate the greedy selection
process we employ an integral representation of the network, based on the
ridgelet transform, that significantly reduces the cardinality of the
dictionary and hence promotes feasibility of the greedy selection. Our approach
allows for the construction of efficient architectures which can be treated
either as improved initializations to be used in place of random-based
alternatives, or as fully-trained networks in certain cases, thus potentially
nullifying the need for backpropagation training. Numerical experiments
demonstrate the tenability of the proposed concept and its advantages compared
to the conventional techniques for selecting architectures and initializations
for neural networks.
