TITLE: Proving $\nabla_A tr(ABA^T C) = CAB + C^T A B^T$
QUESTION [2 upvotes]: The above equation appears without proof on page 9 (equation 3) of Andrew Ng's notes on Machine Learning
I have tried various approaches to prove this to no avail. From the notes it seems that it should be provable from first principles. I tried using the ordinary chain rule on an element by element basis, but this quickly gets unwieldy. Any hint on the approach to resolve this would help a lot.

REPLY [1 votes]: Since the Kronecker inner product on matrices is defined by
$$
 \langle A,B \rangle = \operatorname{tr}(A B^T),
$$
it follows, by the general definition of the gradient, that $\nabla_A \operatorname{tr}(ABA^T C)$ is the unique matrix satisfying
$$
 \operatorname{tr}((A+h)B(A+h)^T C) = \operatorname{tr}(ABA^TC) + \langle \nabla_A \operatorname{tr}(ABA^T C),h\rangle + o(\|h\|)\\ =  \operatorname{tr}(ABA^TC) + \operatorname{tr}(\nabla_A \operatorname{tr}(ABA^T C)h^T)+ o(\|h\|).
$$
However, by repeated application of the cyclic identity for traces and the invariance of the trace under transposition,
$$
 \operatorname{tr}((A+h)B(A+h)^T C) = \operatorname{tr}(ABA^TC + hBA^TC + ABh^TC+hBh^TC)\\
= \operatorname{tr}(ABA^TC) +\operatorname{tr}(hBA^TC) + \operatorname{tr}(ABh^TC) + \operatorname{tr}(hBh^TC)\\
= \operatorname{tr}(ABA^TC) + \operatorname{tr}(C^TAB^Th^T) + \operatorname{tr}(CABh^T) + \operatorname{tr}(hBh^TC)\\
= \operatorname{tr}(ABA^TC) + \operatorname{tr}((C^TAB^T+CAB)h^T) + \operatorname{tr}(hBh^TC),
$$
yielding the desired result.
{"set_name": "stack_exchange", "score": 2, "question_id": 734762}