title: How does the back-propagation algorithm deal with non-differentiable activation functions?
body: <p>While digging through the topic of neural networks and how to efficiently train them, I came across the method of using very simple activation functions, such as the <strong>rectified linear unit</strong> (ReLU), instead of the classic smooth <strong>sigmoids</strong>. The ReLU-function is not differentiable at the origin, so according to my understanding the backpropagation algorithm (BPA) is not suitable for training a neural network with ReLUs, since the chain rule of multivariable calculus refers to smooth functions only.
  However, none of the papers about using ReLUs that I read address this issue. ReLUs seem to be very effective and seem to be used virtually everywhere while not causing any unexpected behavior. Can somebody explain to me why ReLUs can be trained at all via the backpropagation algorithm?</p>