Provable Training of a ReLU Gate with an Iterative Non-Gradient
  Algorithm
  In this work, we demonstrate provable guarantees on the training of a single
ReLU gate in hitherto unexplored regimes. We give a simple iterative stochastic
algorithm that can train a ReLU gate in the realizable setting in linear time
while using significantly milder conditions on the data distribution than
previous such results.
  Leveraging certain additional moment assumptions, we also show a
first-of-its-kind approximate recovery of the true label generating parameters
under an (online) data-poisoning attack on the true labels, while training a
ReLU gate by the same algorithm. Our guarantee is shown to be nearly optimal in
the worst case and its accuracy of recovering the true weight degrades
gracefully with increasing probability of attack and its magnitude.
  For both the realizable and the non-realizable cases as outlined above, our
analysis allows for mini-batching and computes how the convergence time scales
with the mini-batch size. We corroborate our theorems with simulation results
which also bring to light a striking similarity in trajectories between our
algorithm and the popular S.G.D. algorithm - for which similar guarantees as
here are still unknown.
