Data-Driven Learning of Feedforward Neural Networks with Different
  Activation Functions
  This work contributes to the development of a new data-driven method (D-DM)
of feedforward neural networks (FNNs) learning. This method was proposed
recently as a way of improving randomized learning of FNNs by adjusting the
network parameters to the target function fluctuations. The method employs
logistic sigmoid activation functions for hidden nodes. In this study, we
introduce other activation functions, such as bipolar sigmoid, sine function,
saturating linear functions, reLU, and softplus. We derive formulas for their
parameters, i.e. weights and biases. In the simulation study, we evaluate the
performance of FNN data-driven learning with different activation functions.
The results indicate that the sigmoid activation functions perform much better
than others in the approximation of complex, fluctuated target functions.
