# Finding the expected value of the MLE of the multiple linear model

For context,
the linear model is
$$\mathbf{y} = X\mathbf{\beta} + \mathbf{\varepsilon}$$ where $\mathbf{y}$ is the vector of responses $y_i,i=1,2,\ldots,n$, $X$ is the deterministic design matrix $n\times p$ and $\beta$ is the vector of parameters, $p\times 1$.
Also, $\varepsilon$ is the vector of uncorrelated errors with zero mean, $\varepsilon_i,i=1,2,\ldots,n$ where $\mathrm{Var}(\varepsilon_i) = \sigma^2.$

Now, the MLE of $\sigma^2$ in the multiple linear model is
$$\hat{\sigma}^2 := \frac{1}{n} (y-XB)^T (y-XB)$$
And I'm given the formula that for any random $k\times 1$ vector $u$, and constant $k\times k$ matrix $A$, we have

$$\mathbb{E}(u^T A u) = \mathrm{tr}(AV) + \mu^T A \mu$$ where $V=\mathrm{Var}(u), \ \mu = \mathbb{E}(u)$.

So I try to apply this to my MLE.
$$\hat{\sigma}^2 := \frac{1}{n} \varepsilon^T \varepsilon \\ \mathbb{E}(\hat{\sigma}^2) = \frac{1}{n} \mathbb{E}(\varepsilon ^T I_{n\times n} \varepsilon)$$
with $A = I_{n\times n}, u = \varepsilon$.
So $V = \mathrm{Var}(\varepsilon) = \sigma^2 I_{n\times n}, \mu = \mathbb{E}(\varepsilon) = \mathbf{0}$.
hence, $$\mathbb{E}(\hat{\sigma}^2) = \frac{1}{n} ( \mathrm{tr}(I_{n\times n}\sigma^2 I_{n\times n}) = \sigma^2.$$

However, this is incorrect as the solution says that it is $\frac{n-p}{n}\sigma^2$ and it also shows me how it's done. However, I'm not sure where I went wrong.

The problem here is that even tough the error term $\epsilon$ has variance given by $\sigma^2 I_{n\times n}$ the residuals $\hat{\epsilon}$ have a different variance.
$\hat{\sigma}^2=\frac{1}{n}(Y-X\hat\beta)'(Y-X\hat\beta)=\frac{1}{n}\hat{\epsilon}'\hat{\epsilon}$
You should check that, $\hat\epsilon=(I_{n\times n}-P)Y$ where $P$ is the projection matrix $P:=X(X'X)^{-1}X'$. Hence, $Var(\hat\epsilon)=\sigma^2(I_{n\times n}-P)(I_{n\times n}-P)'=(I_{n\times n}-P)$ since $(I_{n\times n}-P)$ is idempotent.
Now, use this as $V$ in your procedure and that should give you the correct result.
The text passage discusses the derivation of the expected value of the Maximum Likelihood Estimator (MLE) of the variance in a multiple linear regression model. It identifies a mistake in the initial approach and provides a corrected method to find the expected value.
single_question_with_answer

the passage contains a single mathematical question regarding the correct expected value of the mle of the variance in a multiple linear regression model, and it provides an answer by correcting the initial approach and explaining the correct method.
Given a text passage, determine:
    1. If it contains a mathematical question (including word problems or calculations)
    2. If an answer to that mathematical question is provided in the text

First, describe the content structure you found in 1-2 sentences.

Then provide your analysis in the following format:
TYPE: <one of the following values>
- NO_QUESTION
- SINGLE_QUESTION
- SINGLE_QUESTION_WITH_ANSWER
- MULTIPLE_QUESTIONS
- MULTIPLE_QUESTIONS_WITH_ANSWERS
system
# Finding the expected value of the MLE of the multiple linear model

For context,
the linear model is
$$\mathbf{y} = X\mathbf{\beta} + \mathbf{\varepsilon}$$ where $\mathbf{y}$ is the vector of responses $y_i,i=1,2,\ldots,n$, $X$ is the deterministic design matrix $n\times p$ and $\beta$ is the vector of parameters, $p\times 1$.
Also, $\varepsilon$ is the vector of uncorrelated errors with zero mean, $\varepsilon_i,i=1,2,\ldots,n$ where $\mathrm{Var}(\varepsilon_i) = \sigma^2.$

Now, the MLE of $\sigma^2$ in the multiple linear model is
$$\hat{\sigma}^2 := \frac{1}{n} (y-XB)^T (y-XB)$$
And I'm given the formula that for any random $k\times 1$ vector $u$, and constant $k\times k$ matrix $A$, we have

$$\mathbb{E}(u^T A u) = \mathrm{tr}(AV) + \mu^T A \mu$$ where $V=\mathrm{Var}(u), \ \mu = \mathbb{E}(u)$.

So I try to apply this to my MLE.
$$\hat{\sigma}^2 := \frac{1}{n} \varepsilon^T \varepsilon \\ \mathbb{E}(\hat{\sigma}^2) = \frac{1}{n} \mathbb{E}(\varepsilon ^T I_{n\times n} \varepsilon)$$
with $A = I_{n\times n}, u = \varepsilon$.
So $V = \mathrm{Var}(\varepsilon) = \sigma^2 I_{n\times n}, \mu = \mathbb{E}(\varepsilon) = \mathbf{0}$.
hence, $$\mathbb{E}(\hat{\sigma}^2) = \frac{1}{n} ( \mathrm{tr}(I_{n\times n}\sigma^2 I_{n\times n}) = \sigma^2.$$

However, this is incorrect as the solution says that it is $\frac{n-p}{n}\sigma^2$ and it also shows me how it's done. However, I'm not sure where I went wrong.

The problem here is that even tough the error term $\epsilon$ has variance given by $\sigma^2 I_{n\times n}$ the residuals $\hat{\epsilon}$ have a different variance.
$\hat{\sigma}^2=\frac{1}{n}(Y-X\hat\beta)'(Y-X\hat\beta)=\frac{1}{n}\hat{\epsilon}'\hat{\epsilon}$
You should check that, $\hat\epsilon=(I_{n\times n}-P)Y$ where $P$ is the projection matrix $P:=X(X'X)^{-1}X'$. Hence, $Var(\hat\epsilon)=\sigma^2(I_{n\times n}-P)(I_{n\times n}-P)'=(I_{n\times n}-P)$ since $(I_{n\times n}-P)$ is idempotent.
Now, use this as $V$ in your procedure and that should give you the correct result.
user
The text passage discusses the derivation of the expected value of the Maximum Likelihood Estimator (MLE) of the variance in a multiple linear regression model. It identifies a mistake in the initial approach and provides a corrected method to find the expected value.

TYPE: SINGLE_QUESTION_WITH_ANSWER

The passage contains a single mathematical question regarding the correct expected value of the MLE of the variance in a multiple linear regression model, and it provides an answer by correcting the initial approach and explaining the correct method.