On Margin Maximization in Linear and ReLU Networks
  The implicit bias of neural networks has been extensively studied in recent
years. Lyu and Li [2019] showed that in homogeneous networks trained with the
exponential or the logistic loss, gradient flow converges to a KKT point of the
max margin problem in the parameter space. However, that leaves open the
question of whether this point will generally be an actual optimum of the max
margin problem. In this paper, we study this question in detail, for several
neural network architectures involving linear and ReLU activations. Perhaps
surprisingly, we show that in many cases, the KKT point is not even a local
optimum of the max margin problem. On the flip side, we identify multiple
settings where a local or global optimum can be guaranteed.
