This is an archived post. You won't be able to vote or comment.

[–] 13 points14 points  (5 children)

Though this does belong under /r/homeworkhelp I'd like to offer some remarks on the 'statistical thinking' behind how to do this.

Let's consider a very simple example. We have some data "Y" we think is explained at least in part by "X"

Since you're in HS maths I won't get too technical, but it should be clear if I rewrite that as Y(X) = something

Now, that would be true if your model perfectly explains the data. But the truth is, it won't always be right. So it's more like: Y(X) = something + error

Now, you probably have more than one data point. So let's say "Y" is a list of y{i}. So something like y{1}, y{2}, ...y{n}. (basically, 'n' data points). Similarly, you have a list of x_{i}.

Let's call that list (well, technically a 'vector') of y{i}'s "Y" and for the x{i}'s, we'll call it "X".

then y{i} = somefunction(x{i}) + error_{i}

somefunction will remain the same, but it takes as input the particular x{i} in question. error{i} is just y{i} - somefunction(x{i})

So let's say X= {1,2,3,4,5,6,7,8,9,10} and Y = {2,4,6,8,10,14,14,16,18,20}

We come up with a linear model to approximate this relationship (I'm not going to teach you how to do that... use google).

we get y{i} = 2.01212*x{i} + 0.13333 + error{i} here, 2.01212*x{i} + 0.13333 is that somefunction(x_{i}), aka the heart of your model. error{i} is simply the difference between the real 'y' value and what your model produces.

In a linear regression we call these errors residuals. To test the fit of your model we generally want to consider how these errors look. We might start by plotting these errors with the x_{i}'s. If they don't look random (follow a distinct pattern) then the model is probably inappropriate. Put simply, in that case the model you specified doesn't properly capture the relationship between X and Y.

So that's one thing to look for. Let's say they look fairly random. The next thing to look at is our model's 'fit'. In a typical simple regression the objective is to minimize these 'errors'. You might think we'd just add all the residuals up and find the model where that sum is minimized. Thing is... we have to take into account that they can be positive or negative. So since we care about the magnitude of the error and not the direction we square them. This 'sum of squared residuals' is an important statistic.

Of course, it's possible to have huge residuals that aren't that significant if your data already deals with large numbers. So we want to take that into account. Therefore we look at how the 'X's and the 'Y's are spread out. For a linear regression, we'll often look at the total squared deviations from the mean for both x's and y's.

Combined with the sum of squared residuals, these measures can be used to produce a 'correlation coefficient'. That will be a direct measure of how good your linear model's fit is.

In summary: Look at how your errors are distributed. A pattern should raise eyebrows. Look at how big the total errors are. Generally, each model will have an accompanying measure of fit. For linear regressions, the pearson correlation coefficient is standard.

[–] 0 points1 point  (0 children)

Very clear explanation, thanks!

[–] 0 points1 point  (3 children)

That's a little more than I asked for, but thank you anyways! A lot of people seem to talk about this sum of squared differences or deviations and I understand that you square them in order to get rid of the negative values, but then why not just use the absolute value and get an average deviation?

[–] 1 point2 points  (0 children)

Wiki explains it well enough hereL http://en.wikipedia.org/wiki/Coefficient_of_determination . If that still doesn't help you, PM. I think I understand what you're looking for, and it's not as complex as it seems..

[–] 0 points1 point  (0 children)

You actually can use absolute deviation, but the sum of squared residuals has nice properties; i.e., it is a differentiable function with a minimum.

[–] 0 points1 point  (0 children)

The coefficient of determination is just that "correlation coefficient" I talked about... squared. In layman's terms, let's say r2 = 0.72. You can interpret is as "72% of the variation in 'y' is explained by 'x'"

As for why we use 'squared differences' instead of 'absolute differences' there are a couple of reasons.

The first is that using squared deviations allows us to obtain errors that are normally distributed. The gaussian/normal distribution has really nice properties that make it ideal over the more exotic distributions you'll have to deal with if you use mean absolute error. Why are they not the same? Because by squaring the errors before adding them together, you are effectively weighting larger errors more.

Suppose your residuals are the following: {-1,7,2,25,8}. For mean absolute error we'd just add 1+7+2+25+8 and divide by 5. But when using squared error measures the '25' will have a much larger impact than the 1,7,2, and 8. This is desirable for a regression.

There is yet another reason we use squared errors over mean absolute error. Absolute values are a pain in the ass to differentiate and integrate in certain situations.

[–] 4 points5 points  (0 children)

There are many ways to evaluate every aspect of a model. At the HS level, probably just the coefficient of determination (R-square), the standard error of estimate, and the F-test are more than enough.

[–] 1 point2 points  (0 children)

Any goodness-of-fit metric would do.

[–] 1 point2 points  (0 children)

You might want to look into cross-validation. Leave-one-out methods.

[–] 0 points1 point  (4 children)

Model selection is a hard problem and the literature is vast. The simplest concept is AIC. I would suggest learning about that.

[–] 1 point2 points  (3 children)

AIC for a single model doesn't say anything about its GoF since it's a relative measure.

[–] 0 points1 point  (2 children)

GoF is not an absolute notion. People might like it to be, but that doesn't make any theoretical sense. No test can prove a model fits the data. All any test can do is prove (as much as statistics can "prove" anything) that no alternative that was considered fits better.

It is true that in very simple situations one can test against a largest possible model. And this is where one gets the idea that goodness of fit is an absolute. But that intuition is only valid when the data are truly multinomial with only a few categories (the original Pearson chi-square setup). When you get more complicated than that, there is no absolution measure of goodness of fit that makes any theoretical sense.

[–] 0 points1 point  (1 child)

Well, I think you're a bit pedantic now. Of course any statistical test is "relative" in the sense that you base the conclusion relative to a null hypothesis that you either reject or fail to reject - at least in the Neyman-Pearson school. There's nothing special about that here.

What is special about AIC is that its numerical value is not absolute. It is not a test and there's no way to quantify how large or small an AIC has to be to make it large or small. If I fit a model and find that its AIC is 280.68, is that a good or bad fit and most importantly: what should I compare it with? This is contrary to statistics for many other GoFs that are defined relative to null hypothesis with known asymptotics.

I don't know what you mean by that there doesn't exist any absolut GoF that makes theoretical sense, and I wonder about your insights into what people might like in that respect and what their intuitions are.

What do you think about the Kolmogorov–Smirnov test? Does that fall under one of your "very simple situations"?

[–] 0 points1 point  (0 children)

It doesn't fall under what I said but Kolmogorov-Smirnov, Cramer-von Mises, Anderson-Darling and the like are all very simple even though nonparametric because they deal only with IID samples from a univariate distribution. No multivariate analogs, no regression-like analogs.

If you want to make AIC (or BIC or Mallow's Cp or SURE or whatever) relative to one particular model just subtract off the AIC (or whatever) for that model. That changes the numbers you report, but doesn't change any inference you do.

I was thinking of situations where AIC is widely used. There are many many models to consider, thousands or millions or more and no clear test of goodness of fit.

Chi-square for contingency tables and Kolmogorov-Smirnov simply do not generalize to complicated situations, not really to any situations more complicated than the ones they work for.

[–] 0 points1 point  (0 children)

When you say 'test' do you just mean 'measure' / 'evaluate' or are you referring explicitly to some form of formal hypothesis test (and if so, why? )?

There are various ways you might measure 'fit' depending on exactly what it's interpreted to mean.

Probably the most basic - but in some ways obvious and often useful - measure would be to look at mean square error - or if the fit doesn't seem to have much bias, perhaps the standard error of the residuals.

But it really depends on what you mean, and what you need.
The provided text passage is an archived Reddit post that explains the concept of linear regression, residuals, and goodness of fit in a statistical model. It does not contain any specific mathematical questions or answers to such questions.
no_question
Given a text passage, determine:
    1. If it contains a mathematical question (including word problems or calculations)
    2. If an answer to that mathematical question is provided in the text

First, describe the content structure you found in 1-2 sentences.

Then provide your analysis in the following format:
TYPE: <one of the following values>
- NO_QUESTION
- SINGLE_QUESTION
- SINGLE_QUESTION_WITH_ANSWER
- MULTIPLE_QUESTIONS
- MULTIPLE_QUESTIONS_WITH_ANSWERS
system
This is an archived post. You won't be able to vote or comment.

[–] 13 points14 points  (5 children)

Though this does belong under /r/homeworkhelp I'd like to offer some remarks on the 'statistical thinking' behind how to do this.

Let's consider a very simple example. We have some data "Y" we think is explained at least in part by "X"

Since you're in HS maths I won't get too technical, but it should be clear if I rewrite that as Y(X) = something

Now, that would be true if your model perfectly explains the data. But the truth is, it won't always be right. So it's more like: Y(X) = something + error

Now, you probably have more than one data point. So let's say "Y" is a list of y{i}. So something like y{1}, y{2}, ...y{n}. (basically, 'n' data points). Similarly, you have a list of x_{i}.

Let's call that list (well, technically a 'vector') of y{i}'s "Y" and for the x{i}'s, we'll call it "X".

then y{i} = somefunction(x{i}) + error_{i}

somefunction will remain the same, but it takes as input the particular x{i} in question. error{i} is just y{i} - somefunction(x{i})

So let's say X= {1,2,3,4,5,6,7,8,9,10} and Y = {2,4,6,8,10,14,14,16,18,20}

We come up with a linear model to approximate this relationship (I'm not going to teach you how to do that... use google).

we get y{i} = 2.01212*x{i} + 0.13333 + error{i} here, 2.01212*x{i} + 0.13333 is that somefunction(x_{i}), aka the heart of your model. error{i} is simply the difference between the real 'y' value and what your model produces.

In a linear regression we call these errors residuals. To test the fit of your model we generally want to consider how these errors look. We might start by plotting these errors with the x_{i}'s. If they don't look random (follow a distinct pattern) then the model is probably inappropriate. Put simply, in that case the model you specified doesn't properly capture the relationship between X and Y.

So that's one thing to look for. Let's say they look fairly random. The next thing to look at is our model's 'fit'. In a typical simple regression the objective is to minimize these 'errors'. You might think we'd just add all the residuals up and find the model where that sum is minimized. Thing is... we have to take into account that they can be positive or negative. So since we care about the magnitude of the error and not the direction we square them. This 'sum of squared residuals' is an important statistic.

Of course, it's possible to have huge residuals that aren't that significant if your data already deals with large numbers. So we want to take that into account. Therefore we look at how the 'X's and the 'Y's are spread out. For a linear regression, we'll often look at the total squared deviations from the mean for both x's and y's.

Combined with the sum of squared residuals, these measures can be used to produce a 'correlation coefficient'. That will be a direct measure of how good your linear model's fit is.

In summary: Look at how your errors are distributed. A pattern should raise eyebrows. Look at how big the total errors are. Generally, each model will have an accompanying measure of fit. For linear regressions, the pearson correlation coefficient is standard.

[–] 0 points1 point  (0 children)

Very clear explanation, thanks!

[–] 0 points1 point  (3 children)

That's a little more than I asked for, but thank you anyways! A lot of people seem to talk about this sum of squared differences or deviations and I understand that you square them in order to get rid of the negative values, but then why not just use the absolute value and get an average deviation?

[–] 1 point2 points  (0 children)

Wiki explains it well enough hereL http://en.wikipedia.org/wiki/Coefficient_of_determination . If that still doesn't help you, PM. I think I understand what you're looking for, and it's not as complex as it seems..

[–] 0 points1 point  (0 children)

You actually can use absolute deviation, but the sum of squared residuals has nice properties; i.e., it is a differentiable function with a minimum.

[–] 0 points1 point  (0 children)

The coefficient of determination is just that "correlation coefficient" I talked about... squared. In layman's terms, let's say r2 = 0.72. You can interpret is as "72% of the variation in 'y' is explained by 'x'"

As for why we use 'squared differences' instead of 'absolute differences' there are a couple of reasons.

The first is that using squared deviations allows us to obtain errors that are normally distributed. The gaussian/normal distribution has really nice properties that make it ideal over the more exotic distributions you'll have to deal with if you use mean absolute error. Why are they not the same? Because by squaring the errors before adding them together, you are effectively weighting larger errors more.

Suppose your residuals are the following: {-1,7,2,25,8}. For mean absolute error we'd just add 1+7+2+25+8 and divide by 5. But when using squared error measures the '25' will have a much larger impact than the 1,7,2, and 8. This is desirable for a regression.

There is yet another reason we use squared errors over mean absolute error. Absolute values are a pain in the ass to differentiate and integrate in certain situations.

[–] 4 points5 points  (0 children)

There are many ways to evaluate every aspect of a model. At the HS level, probably just the coefficient of determination (R-square), the standard error of estimate, and the F-test are more than enough.

[–] 1 point2 points  (0 children)

Any goodness-of-fit metric would do.

[–] 1 point2 points  (0 children)

You might want to look into cross-validation. Leave-one-out methods.

[–] 0 points1 point  (4 children)

Model selection is a hard problem and the literature is vast. The simplest concept is AIC. I would suggest learning about that.

[–] 1 point2 points  (3 children)

AIC for a single model doesn't say anything about its GoF since it's a relative measure.

[–] 0 points1 point  (2 children)

GoF is not an absolute notion. People might like it to be, but that doesn't make any theoretical sense. No test can prove a model fits the data. All any test can do is prove (as much as statistics can "prove" anything) that no alternative that was considered fits better.

It is true that in very simple situations one can test against a largest possible model. And this is where one gets the idea that goodness of fit is an absolute. But that intuition is only valid when the data are truly multinomial with only a few categories (the original Pearson chi-square setup). When you get more complicated than that, there is no absolution measure of goodness of fit that makes any theoretical sense.

[–] 0 points1 point  (1 child)

Well, I think you're a bit pedantic now. Of course any statistical test is "relative" in the sense that you base the conclusion relative to a null hypothesis that you either reject or fail to reject - at least in the Neyman-Pearson school. There's nothing special about that here.

What is special about AIC is that its numerical value is not absolute. It is not a test and there's no way to quantify how large or small an AIC has to be to make it large or small. If I fit a model and find that its AIC is 280.68, is that a good or bad fit and most importantly: what should I compare it with? This is contrary to statistics for many other GoFs that are defined relative to null hypothesis with known asymptotics.

I don't know what you mean by that there doesn't exist any absolut GoF that makes theoretical sense, and I wonder about your insights into what people might like in that respect and what their intuitions are.

What do you think about the Kolmogorov–Smirnov test? Does that fall under one of your "very simple situations"?

[–] 0 points1 point  (0 children)

It doesn't fall under what I said but Kolmogorov-Smirnov, Cramer-von Mises, Anderson-Darling and the like are all very simple even though nonparametric because they deal only with IID samples from a univariate distribution. No multivariate analogs, no regression-like analogs.

If you want to make AIC (or BIC or Mallow's Cp or SURE or whatever) relative to one particular model just subtract off the AIC (or whatever) for that model. That changes the numbers you report, but doesn't change any inference you do.

I was thinking of situations where AIC is widely used. There are many many models to consider, thousands or millions or more and no clear test of goodness of fit.

Chi-square for contingency tables and Kolmogorov-Smirnov simply do not generalize to complicated situations, not really to any situations more complicated than the ones they work for.

[–] 0 points1 point  (0 children)

When you say 'test' do you just mean 'measure' / 'evaluate' or are you referring explicitly to some form of formal hypothesis test (and if so, why? )?

There are various ways you might measure 'fit' depending on exactly what it's interpreted to mean.

Probably the most basic - but in some ways obvious and often useful - measure would be to look at mean square error - or if the fit doesn't seem to have much bias, perhaps the standard error of the residuals.

But it really depends on what you mean, and what you need.
user
The provided text passage is an archived Reddit post that explains the concept of linear regression, residuals, and goodness of fit in a statistical model. It does not contain any specific mathematical questions or answers to such questions.

TYPE: NO_QUESTION