MathGroup Archive 2006

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: Statistical Analysis & Pattern Matching

  • To: mathgroup at
  • Subject: [mg64540] Re: Statistical Analysis & Pattern Matching
  • From: Bill Rowe <readnewsciv at>
  • Date: Mon, 20 Feb 2006 22:31:22 -0500 (EST)
  • Sender: owner-wri-mathgroup at

On 2/20/06 at 6:29 AM, virtualadepts at wrote:

>more often than the others, how do I analyze the data to determine
>which numbers it favors without knowing in advance that it favors any
>of the numbers?

For this particular case a reasonably efficient method would be to generate a table showing each number and the number of times each occurs, i.e.,


>the dice favors any number for sure, but I can assume that it
>favors a number and check to see which numbers it would favor if it

>think of others.  How about an algorithm that generates random
>numbers between 1 and 1,000,000.  Lets say I have a database of 10
>million numbers it has generated, and want to determine what
>numbers it favors.

This is clearly too many values for a simple table to be useful. In fact, the range of the data is too large to make a plot of the histogram useful. So, you would need to compute a meaningful statistic. Which statistic depends on exactly what you want to know.

>likely to produce certain number.

Yes, they are different questions. But depending on exactly what information you want the test used may be the same. The examples you have given above suggest you are always starting with a set of numbers assumed to be drawn from a uniform distribution. So, if I wanted to determine there was at least one number that occurred more often than expected but did not need to identify that number I would likely compute either a chi squared test statistic or the Kolmogorov-Smirnov test statistic. 

For the problem of 10 million random integers between 1 and 1 million, the expected number of times each should occur is 10. So, a chi squared statistic could be computed as

Total[(10-Length@#)^2&/@Split[Sort@data]] + 
100 Length@Complement[Range@10^6,data]

OTOH, if you want to identify the number that occurs most often, then

#1[[Ordering[Last/@#1, -1]]]&@
  ({First@#, Length@#}&/@Split[Sort@data])

will give you the value that occurs most often and the number of times that value occurs. This could be compared with percentage points of the binomial distribution to see if the number of times the value occurred is significantly more (or less) than expected.

Or if you thought there might be several values that occurred more often than they should the most frequent n values with their counts can be extracted with

#1[[Ordering[Last/@#1, -n]]]&@

Alternatively, it may be more useful to find all values that occur more often than a given threshold. This could be done as:

Cases[{First@#, Length@#}&/@Split[Sort@data],{_,_?(#>threshold&)}]

>produce the same number twice, until it has randomly generated
>every other possible number.  Is there a way to predict this is
>happening by looking at the data?  Normally the gamblers fallacy
>isn't a useful idea, but in this case it would help you know in
>advance what the machine will generate because that is how it is

Possibly. For example, a popular method to generate random numbers is a linear congruential generator. And with values created using this method it is possible to compute the parameters of the generator with a sample less than the period of the generator. But, this problem has no general solution. If the algorithm used is well designed, it will not be possible to predict the sequence without first having the output from the entire period. And with a suitably large period, the storage requirements are high enough that there is no practical way to create a program that can predict the next number output from the random number generator being studied.

>How would I solve these problems using mathematica?

Solving the last problem you posed requires a lot more discussion than appropriate for this medium. Hopefully, the comments I made above give you some ideas for solving the other problems you posed.
To reply via email subtract one hundred and four

  • Prev by Date: Re: Statistical Analysis & Pattern Matching
  • Previous by thread: Re: Statistical Analysis & Pattern Matching
  • Next by thread: Re: Step by Step Annotated Derivations