Estimating Multiplicative Relations in Neural Networks
  Universal approximation theorem suggests that a shallow neural network can
approximate any function. The input to neurons at each layer is a weighted sum
of previous layer neurons and then an activation is applied. These activation
functions perform very well when the output is a linear combination of input
data. When trying to learn a function which involves product of input data, the
neural networks tend to overfit the data to approximate the function. In this
paper we will use properties of logarithmic functions to propose a pair of
activation functions which can translate products into linear expression and
learn using backpropagation. We will try to generalize this approach for some
complex arithmetic functions and test the accuracy on a disjoint distribution
with the training set.
