Non-asymptotic estimates for TUSLA algorithm for non-convex learning
  with applications to neural networks with ReLU activation function
  We consider non-convex stochastic optimization problems where the objective
functions have super-linearly growing and discontinuous stochastic gradients.
In such a setting, we provide a non-asymptotic analysis for the tamed
unadjusted stochastic Langevin algorithm (TUSLA) introduced in Lovas et al.
(2021). In particular, we establish non-asymptotic error bounds for the TUSLA
algorithm in Wasserstein-1 and Wasserstein-2 distances. The latter result
enables us to further derive non-asymptotic estimates for the expected excess
risk. To illustrate the applicability of the main results, we consider an
example from transfer learning with ReLU neural networks, which represents a
key paradigm in machine learning. Numerical experiments are presented for the
aforementioned example which supports our theoretical findings. Hence, in this
setting, we demonstrate both theoretically and numerically that the TUSLA
algorithm can solve the optimization problem involving neural networks with
ReLU activation function. Besides, we provide simulation results for synthetic
examples where popular algorithms, e.g. ADAM, AMSGrad, RMSProp, and (vanilla)
SGD, may fail to find the minimizer of the objective functions due to the
super-linear growth and the discontinuity of the corresponding stochastic
gradient, while the TUSLA algorithm converges rapidly to the optimal solution.
