Saturating Auto-Encoders
  We introduce a simple new regularizer for auto-encoders whose hidden-unit
activation functions contain at least one zero-gradient (saturated) region.
This regularizer explicitly encourages activations in the saturated region(s)
of the corresponding activation function. We call these Saturating
Auto-Encoders (SATAE). We show that the saturation regularizer explicitly
limits the SATAE's ability to reconstruct inputs which are not near the data
manifold. Furthermore, we show that a wide variety of features can be learned
when different activation functions are used. Finally, connections are
established with the Contractive and Sparse Auto-Encoders.
