I've been asked to perform a statistical analysis at my work and report on the results. I'm using a 2-sided t-test comparing 2 groups where H0=0 and Ha≠0 at a significance level of .05. For my results my point estimate is .01 and my 95% CI is [-.05, .06]. Clearly my decision is to fail to reject the null and report that there is insufficient evidence to conclude a difference between the groups.

I am presenting this information to people who do not fully understand statistics and who will want the answer to be that there is a difference between the groups. My concern then is that they will interpret the point estimate as showing a .01 difference between the groups and that the confidence interval isn't really that important. I want to emphasize to them how making that conclusion would be incorrect.

My question is, can I alter the confidence interval to emphasize how much the results of the test really center on zero? Does playing with the confidence interval violate the significance level I've established at the beginning of the test? Would this violation jeopardize the entire results?

For instance, I tried different confidence intervals and found that the CI only excludes zero when I have a 20% confidence interval. So, would it be valid for me to say something like, this 20% confidence interval means that we are only 20% sure that the value is not 0?

  • 2
    $\begingroup$ You should look into the concept of confidence distribution, see for instance this post stats.stackexchange.com/questions/133386/… and search this site ... but there are so far no good answers! Look at this new book: amazon.com/… $\endgroup$ – kjetil b halvorsen Mar 2 '17 at 17:46
  • 2
    $\begingroup$ I think people are used to the concept of margin of error from reports of polls. I think a clear non-technical explanation would be more effective than changing confidence intervals. $\endgroup$ – David Lane Mar 2 '17 at 18:54
  • 3
    $\begingroup$ What are the units of measurement? Would 0.01 be a material difference even if it were statistically noticeable? $\endgroup$ – A. Webb Mar 10 '17 at 12:32

One way to do what you ask for, show confidence intervals for (many) different confidence levels at once, is using the concept of a "confidence distribution". This is not well-known and maybe not much used, there is a relatively new book dedicated to the topic: Tore Schweder & Nils Lid Hjort: "Confidence, Likelihood, Probability: Statistical Inference with Confidence distributions" (Cambridge, 2016).

The idea is that you can express the confidence in some "focus parameter" (the parameter you are mainly interested in inference about) as a distribution, to be used much like a bayesian posterior distribution. Then we can "read off" different confidence intervals from that distribution.

In your example, the "focus parameter" will be $\Delta=\mu_1 - \mu_2$. For this example, I will suppose you have equal sample size in the two groups $n=n_1=n_2$ and equal variance $\sigma^2$ which we can estimate by the pooled variance $s_p = \sqrt{\frac{s_1^2+s_2^2}{2}}$. Then inference (about $\Delta$) can be done via the pivot $$ T= \frac{\hat{\Delta}}{s_p \sqrt{2/n}} $$ where $\hat{\Delta}= \bar{x}_1 - \bar{x}_2$ and $T$ is distributed $t_{n+n-2}$, the $t$-distribution with $n+n-2$ degrees of freedom.

I will do the calculations from your data with $n=10$ (you didn't give $n$), with $\hat{\Delta}=0.01$ and the confidence interval as $(-0.04, 0.06)$ which I changed from what you gave so that the estimated $\Delta$ is exactly in the center. There is probably (must be) some rounding error in the numbers you gave.

Now, we can do the same kind of calculations that lead to a confidence interval, but for all confidence levels at the same time, and treating formally $\Delta$ as a random variable. This is the same calculation Fisher did to obtain his fiducial distributions, which, in this example, is the same as what we here call a confidence distribution.

That will give the result: $$ \Delta = \hat{\Delta} - s_p \sqrt{2/n} T $$ where $T$ again has the $t$-distribution with 18 df.

We can present the confidence distribution, either as a confidence density or a confidence cumulative distribution function (or in other ways, not done here). Below I give some R code and the resulting plot:

n  <-  10
s_p  <-  0.05/(qt(0.975, df=18) * sqrt(2/n) )

### The confidence density then becomes
conf_dens   <-  function(d) {
    fac <-  s_p * sqrt(2/n)
    dt( (0.01-d)/fac,df=18 )/fac

conf_dist  <-   function(d) {
    fac <-  s_p * sqrt(2/n)
    pt( (0.01-d)/fac, df=18, lower.tail=FALSE )

plot(conf_dens, from=-0.2, to=+0.22)
plot(conf_dist, from=-0.2, to=0.22)
abline(0.025,0,col="red") ; abline(0.975,0, col="red")
title("Confidence density and distribution", side=3, line=-2, outer=TRUE)

confidence density and confidence cumulative distribution

The two red horizontal lines in the right plot shows how you can read off a 95% confidence interval.


You can alter the confidence interval in order to show that it doesn't support inequality between the groups. In fact, this is what $p$-values do: The $p$-value tells you, that you have to take a $1-100p$%-CI in order to hit the zero with the (in your case: lower) CI bound.

However, as @A. Webb points out in his comment, this is probably not the best to do. If some people in your audience understand a bit about statistics and still want a significant result, they would suggest to repeat the experiment with much larger sample sizes. As the t-test is consistent, one will eventually find a significant non-zero difference even if it is smaller than $0.01$.

So you should rather point out (if you can) that the estimated difference is too small to be relevant. This will not be influenced by sample sizes and statistical power. Instead, it is more bound to the underlying fact and so it will be more convincing your audience lacking statistical knowledge. Tactically (or if you don't have an idea), you ask them first for their judgement of what a relevant difference might be.


Your Answer

