TITLE: Show that if $||Y - X_j\hat{\beta}_j||^2_2 < ||Y - X_{j^*}\hat{\beta}_{j*}||^2_2$ then $X_j^TY/||X_j||_2 \geq \pm X_{j*}^TY/||X_{j*}||_2$
QUESTION [0 upvotes]: Suppose $Y \in \mathbb{R}^n$, $X \in \mathbb{R}^{n \times p}$, $\beta \in \mathbb{R}^p$ and $\epsilon \in \mathbb{R}^n$ with mean 0 and covariance $\sigma^2I_n$.  Assume the following relationship between $X$ and $Y$:
$$Y = X\beta + \epsilon$$
Suppose that $p=3$, and $X_j \in \mathbb{R}^n$ is the $j$th column of the matrix $X$ for $j = 1, 2, 3$. Let $\hat{\beta}_j = \left(X_j^TX_j\right)^{-1}X_j^TY$ be the least squares estimator.
I want to verify the following statement:
"If $\hat{\beta}_3$ has the lowest residual sum of error, i.e.,
$$||Y - X_3\hat{\beta}_3||^2_2 <  ||Y - X_2\hat{\beta}_2||^2_2 \text{ and } ||Y - X_3\hat{\beta}_3||^2_2 <  ||Y - X_1\hat{\beta}_1||^2_2,$$ then $$s_3X_3^TY/||X_3||_2 \geq \pm X_1^TY/||X_1||_2 \text{ and } s_3X_3^TY/||X_3||_2 \geq \pm X_2^TY/||X_2||_2,"$$
where $s_3 = sign(X_3^TY)$.
My thought process is as follows:
Since $X_j$ is a vector, then $\hat{\beta}_j = \left(X_j^TX_j\right)^{-1}X_j^TY = X_j^TY/||X_j||^2_2$. However, I'm not sure where to go from here.

REPLY [0 votes]: Let $Z_j:=X_j^{\top}Y/\|X_j\|_2$. Then
$$
\|Y-X_j\hat{\beta}_j\|_2^2=Y^{\top}(I-X_j(X_j^{\top}X_j)^{-1}X_j^{\top})Y=\|Y\|_2^2-Z_j^2,
$$
and
$$
\|Y-X_3\hat{\beta}_3\|_2^2\le \|Y-X_j\hat{\beta}_j\|_2^2
$$
is equivalent to
$$
Z_3^2\ge Z_j^2 \quad\Leftrightarrow\quad |Z_3|\ge |Z_j|.
$$
{"set_name": "stack_exchange", "score": 0, "question_id": 3991899}