system
Given the following post, respond as if you're commenting on Reddit:
human
What conclusion can be drawn from statistically insignificant results? (Layman question and text provided)

Hello guys, I just wanted to ask you if somebody could give me very 
basic (layman) explanation about one question.
If for example you did an experiment on yourself where you tried a drug to see if it helps you with your condition, you took it 100.000 times but it only helped you 3 times. What conclusion can you draw from this? Is the result statistically not significant and therefore the hypothesis that a drug helps you can be rejected (so you can conclude that this drug doesn't help you). I know this is a very lame example, but try to give me an explanation for this particular scenario. It seems logical for me to conclude that a drug doesn't help, but I want to see it backed up by science. Thanks in advance!
gpt
The example you gave is not quite enough to explain the meaning of statistical significance. You took the drug 100 times and it helped you 3 times so it worked 3% of the time and there is not much else to say about it.

To use statistical significance you have to compare the drug to something. So you would also have to take a placebo 100 times. Let's say the placebo "helped" (you got better) 1 time. Now you have 2 sets of figures: drug worked 3% of the time, placebo worked 1% of the time - and with this you can do some maths.

Basically the question statistical significance attempts to answer is: is this 3% vs 1% difference due to chance, or to the drug actually having some effect?

Now we have to back up a bit and explain what is meant by "chance". "Chance" arises because you cannot take the drug (or the placebo) every day forever. If you could, by some stretch of the imagination, take the drug every day forever and find that it worked (say) 6% of the time, and then (maybe in a parallel universe) take the placebo every day forever and find that it worked 4% of the time, there would be no need for statistical analysis: the drug works better, 6% vs 4%, no further calculation required.

Unfortunately you can't do that - we have to analyze the 100 times you took the drug and the 100 times you took the placebo - and so, statistical analysis.

The "by chance" refers to the fact that the 100 times you took the drug and the 100 times you took the placebo can be considered *samples* of the situation in which you took the drug or placebo everyday forever. 

Let's say that the "actual" response rate when taking the drug everyday forever is 6% (NB we have no way of knowing the "actual" figure, this is just for illustration) - a sample of 100 from this scenario could produce a response rate of 5% or 6% or 4 % or 3% or 2% or indeed any number. It is more likely to be close to the "actual" 6% than not, but it *could* be anything from 0 to 100%. In short, the response rate in your sample of 100 is affected by chance, because of the fact that you can only take samples, and never know the "actual" response rate.

What statistical significance does is this: it assumes that the "actual" (everyday forever) response rate to drug and placebo are *the same*, then it asks *what is the probability p that the figures we got (3% vs 1%) are purely because of the fact that we are restricted to taking samples of 100?*

The maths will then crank out a figure - say p=0.06. This means that the probability that you got the 3% and 1% figures despite the "actual" figures being equal is 0.06. Before turning the maths crank you should have already decided on what "p" you would consider statistically significant. Let's say it was p=0.05. That being the case, the figures show that there is no statistically significant difference between the drug and the placebo. In other words you have concluded that the figures your experiment has produced have come about by chance and the *"actual" figures for drug and placebo could be identical*. 

TL;DR: "Statistically not significant" means that the figures your experiment produced could have come about by the fact you that have to sample, rather than from an "actual" difference between the the two treatments.