Continuously Differentiable Exponential Linear Units
  Exponential Linear Units (ELUs) are a useful rectifier for constructing deep
learning architectures, as they may speed up and otherwise improve learning by
virtue of not have vanishing gradients and by having mean activations near
zero. However, the ELU activation as parametrized in [1] is not continuously
differentiable with respect to its input when the shape parameter alpha is not
equal to 1. We present an alternative parametrization which is C1 continuous
for all values of alpha, making the rectifier easier to reason about and making
alpha easier to tune. This alternative parametrization has several other useful
properties that the original parametrization of ELU does not: 1) its derivative
with respect to x is bounded, 2) it contains both the linear transfer function
and ReLU as special cases, and 3) it is scale-similar with respect to alpha.
