title: Numpy transformation to normal distribution
body: <p>I have an array of data. I checked if it was normally distributed:</p>
  
  <pre><code>import sys
  import scipy
  from scipy import stats
  from scipy.stats import mstats
  from scipy.stats import normaltest
  
  Data = []
  for line in open(sys.argv[1]):
      line = line.strip()
      Data.append(float(line))
  print scipy.stats.normaltest(Data)
  </code></pre>
  
  <p>The output was: (36.444648754208075, 1.2193968690198398e-08)</p>
  
  <p>Then, I wrote a small script to normalise the data:</p>
  
  <pre><code>import sys
  import numpy as np
  fileopen = open(sys.argv[1])
  UntransformedArray = []
  for line in fileopen:
      line = float(line.strip())
      UntransformedArray.append(line)
  TransformedArray = (UntransformedArray - np.mean(UntransformedArray)/np.std(UntransformedArray))
  NewList = TransformedArray.tolist()
  for i in NewList:
      print i
  </code></pre>
  
  <p>And then I checked for normality again using the first script and the output was 
  (36.444648754209595, 1.2193968690189117e-08).</p>
  
  <p>...the same as the previous score, and not normally distributed.</p>
  
  <p>is one of my scripts wrong?</p>
  
  <p>Also, should I mention that the average of my data is 0.056, the numbers range from 0.014 to 0.171 (85 observations), I'm not sure if the fact that the numbers are so small matters. </p>
  
  <p>A sample of the untransformed and transformed data:</p>
  
  <p>Untransformed:</p>
  
  <pre><code>0.055
  0.074
  0.049
  0.067
  0.038
  0.037
  0.045
  0.041
  </code></pre>
  
  <p>Transformed data:</p>
  
  <pre><code>-2.13696814254
  -2.11796814254
  -2.14296814254
  -2.12496814254
  -2.15396814254
  -2.15496814254
  -2.14696814254
  </code></pre>
  
  <p>Edit 1:</p>
  
  <p>When I edit the code slightly to account for parenthesis being in the wrong place:</p>
  
  <pre><code>TransformedMean = (UntransformedArray - np.mean(UntransformedArray))
  TransformedArray = (TransformedMean/np.std(UntransformedArray))
  NewList = TransformedArray.tolist()
  for i in NewList:
      print i
  </code></pre>
  
  <p>The output I get it different:</p>
  
  <p>Example:</p>
  
  <pre><code>-0.0385683544143
  0.705333390576
  -0.273484694937
  0.431264326632
  -0.704164652563
  -0.743317375984
  </code></pre>
  
  <p>However, when I check for normality:
  (36.444648754241328, 1.2193968689995659e-08)</p>
  
  <p>It is still not normally distributed (and is still the exact same score as the other times)?</p>
  
  <p>Edit 2: </p>
  
  <p>I then tried a different method of normalising the data:</p>
  
  <pre><code>import sys
  import scipy
  from scipy import stats
  from scipy.stats import boxcox
  
  Data = [(float(line.strip())) for line in open(sys.argv[1])]
  scipy.stats.boxcox(Data)
  </code></pre>
  
  <p>I get the error: TypeError: unsupported operand type(s) for ** or pow(): 'list' and 'float'</p>
  
  <p>EDIT 3: Due to comment from user, the problem was understanding the difference in normalising values, versus normalising a distribution.</p>
  
  <p>Edited code:</p>
  
  <pre><code>import sys
  import numpy as np
  
  fileopen = open(sys.argv[1])
  UntransformedArray = []
  for line in fileopen:
      line = float(line.strip())
      UntransformedArray.append(line)
  
  List1 =  np.log(UntransformedArray) 
  for i in List1:
      print i
  </code></pre>
  
  <p>Checking for normalisation:
  (4.0435072214905938, 0.13242304287973003)</p>
  
  <p>(works in this case, depending on skewness of the data).</p>
  
  <p>Edit 4: Or using a BoxCox transformation:</p>
  
  <pre><code>import sys
  import scipy
  from scipy import stats
  from scipy.stats import boxcox
  import numpy as np
  
  Data = []
  for line in open(sys.argv[1]):
      line = line.strip()
      Data.append(float(line))
  
  data = scipy.stats.boxcox(np.array(Data))
  for i in data[0]:
      print i
  </code></pre>
  
  <p>Check for normalisation: (2.9085877478631956, 0.23356523218452238)</p>