Concavifiability and convergence: necessary and sufficient conditions
  for gradient descent analysis
  Convergence of the gradient descent algorithm has been attracting renewed
interest due to its utility in deep learning applications. Even as multiple
variants of gradient descent were proposed, the assumption that the gradient of
the objective is Lipschitz continuous remained an integral part of the analysis
until recently. In this work, we look at convergence analysis by focusing on a
property that we term as concavifiability, instead of Lipschitz continuity of
gradients. We show that concavifiability is a necessary and sufficient
condition to satisfy the upper quadratic approximation which is key in proving
that the objective function decreases after every gradient descent update. We
also show that any gradient Lipschitz function satisfies concavifiability. A
constant known as the concavifier analogous to the gradient Lipschitz constant
is derived which is indicative of the optimal step size. As an application, we
demonstrate the utility of finding the concavifier the in convergence of
gradient descent through an example inspired by neural networks. We derive
bounds on the concavifier to obtain a fixed step size for a single hidden layer
ReLU network.
