Learning a Single Neuron with Bias Using Gradient Descent
  We theoretically study the fundamental problem of learning a single neuron
with a bias term ($\mathbf{x} \mapsto \sigma(<\mathbf{w},\mathbf{x}> + b)$) in
the realizable setting with the ReLU activation, using gradient descent.
Perhaps surprisingly, we show that this is a significantly different and more
challenging problem than the bias-less case (which was the focus of previous
works on single neurons), both in terms of the optimization geometry as well as
the ability of gradient methods to succeed in some scenarios. We provide a
detailed study of this problem, characterizing the critical points of the
objective, demonstrating failure cases, and providing positive convergence
guarantees under different sets of assumptions. To prove our results, we
develop some tools which may be of independent interest, and improve previous
results on learning single neurons.
