Learning a Single Neuron for Non-monotonic Activation Functions
  We study the problem of learning a single neuron $\mathbf{x}\mapsto
\sigma(\mathbf{w}^T\mathbf{x})$ with gradient descent (GD). All the existing
positive results are limited to the case where $\sigma$ is monotonic. However,
it is recently observed that non-monotonic activation functions outperform the
traditional monotonic ones in many applications. To fill this gap, we establish
learnability without assuming monotonicity. Specifically, when the input
distribution is the standard Gaussian, we show that mild conditions on $\sigma$
(e.g., $\sigma$ has a dominating linear part) are sufficient to guarantee the
learnability in polynomial time and polynomial samples. Moreover, with a
stronger assumption on the activation function, the condition of input
distribution can be relaxed to a non-degeneracy of the marginal distribution.
We remark that our conditions on $\sigma$ are satisfied by practical
non-monotonic activation functions, such as SiLU/Swish and GELU. We also
discuss how our positive results are related to existing negative results on
training two-layer neural networks.
