TITLE: Chain rule for matrices
QUESTION [1 upvotes]: I trying to work through derivation of Ordinary Least Squares for Linear Regression and I'm stumbling over taking the partial derivative of this term $$\frac{\partial (\beta^TX^TX\beta)}{\partial\beta}$$ I found this post so I can see the derivative should be $$X^TX\beta + (\beta X^TX)^T = 2X^TX\beta$$ but I still don't see why how the chain rule was used to get the derivative for $\beta$. Where does the addition come from and how is the chain rule for matrices different from the chain rule for regular variables?

REPLY [0 votes]: To find the gradient of $f(x)=(1/2)\|Ax-b\|^2$, note that $f(x)=g(h(x))$ where $h(x) =Ax$ and $g(y)=(1/2) \|y-b\|^2$. By the chain rule, $f'(x)=g'(h(x))h'(x)$. Clearly $h'(x)=A$ and $g'(y) = (y-b)^T$. Thus, we have $f'(x)=(Ax-b)^TA$. If we use the convention that $\nabla f(x) = f'(x)^T$, it follows that $\nabla f(x) = A^T(Ax-b)$.
{"set_name": "stack_exchange", "score": 1, "question_id": 2423216}