Density-embedding layers: a general framework for adaptive receptive
  fields
  The effectiveness and performance of artificial neural networks, particularly
for visual tasks, depends in crucial ways on the receptive field of neurons.
The receptive field itself depends on the interplay between several
architectural aspects, including sparsity, pooling, and activation functions.
In recent literature there are several ad hoc proposals trying to make
receptive fields more flexible and adaptive to data. For instance, different
parameterizations of convolutional and pooling layers have been proposed to
increase their adaptivity. In this paper, we propose the novel theoretical
framework of density-embedded layers, generalizing the transformation
represented by a neuron. Specifically, the affine transformation applied on the
input is replaced by a scalar product of the input, suitably represented as a
piecewise constant function, with a density function associated with the
neuron. This density is shown to describe directly the receptive field of the
neuron. Crucially, by suitably representing such a density as a linear
combination of a parametric family of functions, we can efficiently train the
densities by means of any automatic differentiation system, making it adaptable
to the problem at hand, and computationally efficient to evaluate. This
framework captures and generalizes recent methods, allowing a fine tuning of
the receptive field. In the paper, we define some novel layers and we
experimentally validate them on the classic MNIST dataset.
