TITLE: cycles in gradient descent on non-euclidean manifold?
QUESTION [3 upvotes]: In a euclidean space, we cannot have gradient descent of a function $f$ and still have cycles. i.e. if $x(t)$ is the path traced by the dynamical system given by $\dot x(t)=-\nabla_x f(x)$, for some function $f$, then we cannot have $x(t_0)=x(t_1)\neq x(s)$ for $t_0>s>t_1$.
My question is whether there is an example of an (possibly exotic) space where this doesn't hold. Intuitively I would say it should always hold, but I might not be imagining a weird space where it doesn't.

REPLY [4 votes]: Let $\langle,\rangle$ be a bilinear form on $\mathbb R^n$, not necessarily positive definite, and define the associated gradient of $f\in C^1$ as the unique vector field $\nabla f(x)$ such that$^{[1]}$
$$
\langle \nabla f(x), v\rangle = \frac{\partial}{\partial \epsilon} f(x+\epsilon v)\Big|_{\epsilon=0}, \qquad \forall v\in \mathbb R^n.$$
If $\langle,\rangle$ is not positive definite, then 

the corresponding gradient flow can have cycles (i.e., periodic solutions).

For a concrete example take the bilinear form on $\mathbb R^2$
$$
\langle (x_1, x_2), (y_1, y_2)\rangle:=x_1y_1-x_2y_2.$$ 
The corresponding gradient is the vector field 
$$
\nabla f=(\partial_{x_1} f, - \partial_{x_2} f);$$
thus, the associated gradient flow $\dot u =-\nabla u$ is
$$
\begin{cases} \partial_t u_1 = -\partial_1f( u_1, u_2),\\ \partial_t u_2= \partial_2 f(u_1,u_2).\end{cases} 
$$
Taking $f(x_1, x_2)=x_1x_2$ yields 
$$
\begin{cases} \partial_t u_1 = -u_2,\\ \partial_t u_2= u_1,\end{cases} 
$$
whose general solution is $(u_1, u_2)=(R\cos (t+\phi), R\sin (t+\phi))$, for arbitrary $R\ge 0$ and $\phi\in\mathbb R$. In particular, all solutions are cycles.

$[1]$. This requires that $\langle,\rangle$ be non-degenerate, which means that the equation $\langle x, y\rangle=0$ can hold for all $y\in\mathbb R^n$ if and only if $x=0$. Notice, moreover, that this is a special case of the standard definition of gradient, in the context of Riemannian and pseudo-Riemannian manifolds. If the bilinear form is the standard dot product, this definition produces the ordinary gradient of vector calculus.
{"set_name": "stack_exchange", "score": 3, "question_id": 3099636}