TITLE: How to get gradient of linear equations?
QUESTION [0 upvotes]: I am new to derivatives of linear equation systems. And I would like to 
$$\|Ax-b\|^2 = (Ax-b)^T(Ax-b) (equation)$$
And how do I reach to this gradient: $\Delta_x = 2(Ax-b)^TA$?
If I develop the equation I get: $Ax^TAx-b^TAx-Ax^Tb+b$. But how do I get to the gradient ?

REPLY [0 votes]: This question comes up often, and the most elegant solution uses the chain rule. Let $f(x) = \| Ax - b ||_2^2$. Note that $f(x) = g(h(x))$, where $h(x) = Ax - b$ and $g(u) = \| u \|^2$. The derivatives of $h$ and $g$ are $h'(x) = A$ and $g'(u) = 2u^T$ (a row vector). By the chain rule,
$$
f'(x) = g'(h(x)) h'(x) = 2(Ax - b)^T A.
$$
{"set_name": "stack_exchange", "score": 0, "question_id": 3557185}