Self-Organized Operational Neural Networks with Generative Neurons
  Operational Neural Networks (ONNs) have recently been proposed to address the
well-known limitations and drawbacks of conventional Convolutional Neural
Networks (CNNs) such as network homogeneity with the sole linear neuron model.
ONNs are heterogenous networks with a generalized neuron model that can
encapsulate any set of non-linear operators to boost diversity and to learn
highly complex and multi-modal functions or spaces with minimal network
complexity and training data. However, Greedy Iterative Search (GIS) method,
which is the search method used to find optimal operators in ONNs takes many
training sessions to find a single operator set per layer. This is not only
computationally demanding, but the network heterogeneity is also limited since
the same set of operators will then be used for all neurons in each layer.
Moreover, the performance of ONNs directly depends on the operator set library
used, which introduces a certain risk of performance degradation especially
when the optimal operator set required for a particular task is missing from
the library. In order to address these issues and achieve an ultimate
heterogeneity level to boost the network diversity along with computational
efficiency, in this study we propose Self-organized ONNs (Self-ONNs) with
generative neurons that have the ability to adapt (optimize) the nodal operator
of each connection during the training process. Therefore, Self-ONNs can have
an utmost heterogeneity level required by the learning problem at hand.
Moreover, this ability voids the need of having a fixed operator set library
and the prior operator search within the library in order to find the best
possible set of operators. We further formulate the training method to
back-propagate the error through the operational layers of Self-ONNs.
