Faster Optimization on Sparse Graphs via Neural Reparametrization
In mathematical optimization, second-order Newton's methods generally
converge faster than first-order methods, but they require the inverse of the
Hessian, hence are computationally expensive. However, we discover that on
sparse graphs, graph neural networks (GNN) can implement an efficient
Quasi-Newton method that can speed up optimization by a factor of 10-100x. Our
method, neural reparametrization, modifies the optimization parameters as the
output of a GNN to reshape the optimization landscape. Using a precomputed
Hessian as the propagation rule, the GNN can effectively utilize the
second-order information, reaching a similar effect as adaptive gradient
methods. As our method solves optimization through architecture design, it can
be used in conjunction with any optimizers such as Adam and RMSProp. We show
the application of our method on scientifically relevant problems including
heat diffusion, synchronization and persistent homology.