{'fraction_non_alphanumeric': 0.02707383773928897, 'fraction_numerical': 0.012579762989972652, 'mean_word_length': 5.265562535693889, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A Statistical Significance Simulation Study for the General Scientist

When a scientist performs an experiment they normally acquire a set of measurements and are expected to demonstrate that their results are"statistically significant"thus confirming whatever hypothesis they are testing. The main method for establishing statistical significance involves demonstrating that there is a low probability that the observed experimental results were the product of random chance. This is typically defined as p<0.05, which indicates there is less than a 5% chance that the observed results occurred randomly. This research study visually demonstrates that the commonly used definition for"statistical significance"can erroneously imply a significant finding. This is demonstrated by generating random Gaussian noise data and analyzing that data using statistical testing based on the established two-sample t-test. This study demonstrates that insignificant yet"statistically significant"findings are possible at moderately large sample sizes which are very common in many fields of modern science.


Introduction
Establishing statistical significance is extremely common among scientists. This involves demonstrating that there is a low probability that a scientist's observed measurements were the result of random chance. This is typically defined as (p<0.05) which indicates that there is less than a 5% chance that the observed differences were the result of randomness. Statistical significance can be established using a wide variety of statistical tests which compare a scientist's measurements with randomly generated distributions to determine a p-value (from which statistical significance is established). It is known that as the number of samples increases, the amount of difference needed between our two distributions to obtain statistical significance (p<.05) gets smaller. The main focus of this research paper is to present data which demonstrates that as the number of samples becomes large, the amount of separation between our two groups needed to obtain 'statistical significance' becomes negligible. This effect indicates that scientists have a potentially extremely low threshold for obtaining statistical significance. In its most extreme form, a "statistically significant" effect is in fact qualitatively insignificant.
In this study we have elected to perform statistical testing using the widely accepted and established two-sample t-test [1]. It should be noted that the t-test was developed in a beer factory in 1908 over one-hundred years ago by a scientist writing under a false name (Student). This was long before the advent of computers, thus long before a scientist had the ability to perform statistical testing on groups of data with large numbers of samples. In fact, the original introduction of the t-test [1] provided look-up tables to assist in statistical computations that allow the researcher to perform analyses on data groups with up to only 10 samples. In those days it was unreasonable for someone to manually compute p-values on hundreds or thousands of samples. In the present research environment a journal paper reviewer is likely to require many more than 10 samples from a typical scientist's experiment, thus inadvertently lowering the bar for obtaining the desired "statistical significance". After reading this study it should be clear that the ttest was developed for another era and alternative techniques would benefit the modern scientist. Or as Bill Rozeboom wrote in 1960 "The statistical folkways of a more primitive past continue to dominate the local scene" [2]. Rozeboom was writing about problems with statistical testing over 50 years after the t-test was first created. It is now 50 years after Rozeboom wrote this commentary and his words are still as relevant as ever. There have been many critiques of how statistical significance testing and null hypothesis testing is used [2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20], yet despite the many shortcomings highlighted, performing hypothesis testing based on a p-value threshold (p<.05) is still one of the most common statistical techniques used by modern scientists.
Standard thought has it that if we increase our number of samples then the computed statistical p-value becomes more and more reliable. In fact, as we add more and more samples, the amount of separation needed between our groups to achieve statistical significance gets smaller. This is because the p-value computations are based on random data. Once the number of samples becomes very large, the amount of overlap observed between large randomly generated distributions will always be large, leading to very little separation required between the two distributions to achieve a p-value below 0.05. Or put another way, we have a threshold for statistical significance that is so low that (as long as we have an adequate number of samples) all we need is to have two noisy signals that are ever so marginally dissimilar in order to achieve "statistical significance". This low threshold for achieving statistical significance has the potential to greatly affect a scientist's approach to their experiments. As scientists, our career prospects (and thus our prestige and personal finances) are heavily dependent on our accumulation of peer-reviewed journal papers. This personal motivation biases us towards getting our research accepted for publication. Since it is extremely common for a journal paper reviewer to require that our experimental results be tested for statistical significance, we are generally biased towards finding statistical significance in our experiments in order to accumulate journal publications and to succeed in our careers.
The word 'significant' is qualitative and subjective. Whether something is 'significant' is in the eye of the beholder. When we add the word 'statistics', we add a strong quantitative word to the very qualitative word 'significant'. This lends an appearance of credibility and certainty to any experiment that achieves a p-value below 0.05, simply because this is the widely accepted threshold for achieving 'statistical significance'.
Since statistical significance is based on random distributions, performing hypothesis testing on the p-value calculation (p<.05) is like asking the question: did our experiment do better than 95% of randomness? But since the vast majority of scientists are likely to have constructed their experiments in a somewhat logical manner, they are generally liable to do at least a little better than random chance. Thus scientists are highly likely to find statistical significance in their experiments, especially if they perform their experiments with many samples. This study is designed to visually illustrate that achieving statistical significance (p<.05) at moderately large sample sizes requires only marginally significant (or possibly even insignificant) experimental data.

Methods
P-values are computed from lookup tables which are created from randomly generated distributions of data. This research study's methods are designed to visually illustrate how much separation is required between two groups of numbers in order to achieve the standard definition of statistical significance (p<.05) at a variety of sample sizes. This is accomplished by generating large amounts of normal (Gaussian) random distributions.
We have elected to perform our analysis on two-sample tests where two groups of numbers are compared with each other in order to determine if they are statistically significantly different from each other. This is one of the most pervasive types of statistical testing as it is extremely common for a scientist to compare two groups of numbers (for example an experimental group and a control group). For this study 1000 pairs of random distributions were created at each example sample size. Of all the randomly generated cases, the pair that exhibit the highest p-value below 0.05 is selected for presentation as a visual example of how much separation is needed between two groups of data in order to achieve 'statistical significance' at the given sample size.
Random distributions were generated across a wide variety of group sample sizes where the image's dimensions are expressed as a factor of 2 (4, 16, 64, 256, 1024, 4096, 16384, noise pairs demonstrates how the amount of separation between two barely statistically significantly different groups changes as the number of samples is varied. All statistical significance testing was performed using one of the most common statistical tests available, the two-sample t-test. This was selected so that our statistical testing method matches the type of distributions being randomly generated (Gaussian noise / normal distributions). In addition, for each sample size setting, the number of randomly created distributions that have a p-value below 0.05 are enumerated. All random normal (Gaussian) distributions were created using the mathematical and statistical package Matlab (Mathworks, Natick, MA, USA). Statistical testing was performed with the established two-sample t-test provided in Matlab.

Results
Pairs of randomly generated distributions with p-values just below 0.05 are included as the main results of this study.  Table 1 also presents the total number of randomly generated distributions which achieved a statistically significant difference as the term is typically defined (p<.05), using the popular and well established two-sample t-test. Since the experiment involves creating 1000 randomized distributions we expect to find 50 (5%) of those samples being statistically significant (p<.05). The results from each trial were confirmed to be close to 50 samples achieving statistical significance out of each 1000 randomly created cases.
When examining each noise image pair, a scientist can interpret the two visual image distributions as being very close to the threshold for obtaining statistical significance at the given number of samples. Note that the difference between two statistically significantly different distributions gets smaller as the number of samples increases.

Discussion
It can be seen from the results that at a p-value just below 0.05, the two randomly Data was also included demonstrating that approximately 5% of the randomly generated samples created for this experiment achieve statistical significance (as the term is typically defined p <.05). This is presented in the final column of This study's findings are potentially of broad interest to scientists in general. Establishing statistical significance with a p-value provides us with an answer to the question "did we beat 95% of randomness?" But randomness is a very low bar to set for ourselves, thus ensuring that scientists who work with reasonably large sample sizes will be able to go on finding statistically significant (p<.05) results (almost) wherever they look for them.