A Use of Even Activation Functions in Neural Networks
  Despite broad interest in applying deep learning techniques to scientific
discovery, learning interpretable formulas that accurately describe scientific
data is very challenging because of the vast landscape of possible functions
and the "black box" nature of deep neural networks. The key to success is to
effectively integrate existing knowledge or hypotheses about the underlying
structure of the data into the architecture of deep learning models to guide
machine learning. Currently, such integration is commonly done through
customization of the loss functions. Here we propose an alternative approach to
integrate existing knowledge or hypotheses of data structure by constructing
custom activation functions that reflect this structure. Specifically, we study
a common case when the multivariate target function $f$ to be learned from the
data is partially exchangeable, \emph{i.e.} $f(u,v,w)=f(v,u,w)$ for $u,v\in
\mathbb{R}^d$. For instance, these conditions are satisfied for the
classification of images that is invariant under left-right flipping. Through
theoretical proof and experimental verification, we show that using an even
activation function in one of the fully connected layers improves neural
network performance. In our experimental 9-dimensional regression problems,
replacing one of the non-symmetric activation functions with the designated
"Seagull" activation function $\log(1+x^2)$ results in substantial improvement
in network performance. Surprisingly, even activation functions are seldom used
in neural networks. Our results suggest that customized activation functions
have great potential in neural networks.
