title: Handling NaNs in linear regression -- scipy?
body: <p>I have a data set with <code>NaN</code>s scattered amongst the data. I'm using pandas to extract the data from a file, and numpy to process it. This is my code for reading the data:</p>
  
  <pre><code>import pandas as pd
  import numpy as np
  
  def makeArray(band):
      """
      Takes as argument a string as the name of a wavelength band.
      Converts the list of magnitudes in that band into a numpy array,
      replacing invalid values (where invalid == -999) with NaNs.
      Returns the array.
      """
      array_name = band + '_mag'
      array = np.array(df[array_name])
      array[array==-999]=np.nan
      return array
  
  #   Read data file
  fields = ['no', 'NED', 'z', 'obj_type','S_21', 'power', 'SI_flag', 
            'U_mag', 'B_mag', 'V_mag', 'R_mag', 'K_mag', 'W1_mag',
            'W2_mag', 'W3_mag', 'W4_mag', 'L_UV', 'Q', 'flag_uv']
  
  magnitudes = ['U_mag', 'B_mag', 'V_mag', 'R_mag', 'K_mag', 'W1_mag',
            'W2_mag', 'W3_mag', 'W4_mag']
  
  df = pd.read_csv('todo.dat', sep = ' ',
                     names = fields, index_col = False)
  
  #   Define axes for processing
  redshifts = np.array(df['z'])
  y = np.log(makeArray('K'))
  mask = np.isnan(y)
  </code></pre>
  
  <p>I guess a minimal working example would be:</p>
  
  <pre><code>import numpy as np
  import matplotlib.pyplot as plt
  from scipy import stats
  
  randomNumberGenerator = np.random.RandomState(1000)
  x = 4 * randomNumberGenerator.rand(100)
  y = 4 * x - 1+ randomNumberGenerator.randn(100)
  y[50] = np.nan
  
  slope, intercept, r_value, p_value, std_err = stats.linregress(x,y)
  fit = slope*x + intercept
  
  plt.scatter(x, y)
  plt.plot(x, fit)
  plt.show()
  </code></pre>
  
  <p>Commenting out the <code>y[50] = np.nan</code> line in the MWE produces a nice graph, but including it produces the same error message as with my actual data:</p>
  
  <pre><code>C:\Users\Jeremy\Anaconda3\lib\site-packages\scipy\stats\_distn_infrastructure.py:879: RuntimeWarning: invalid value encountered in greater
    return (self.a &lt; x) &amp; (x &lt; self.b)
  C:\Users\Jeremy\Anaconda3\lib\site-packages\scipy\stats\_distn_infrastructure.py:879: RuntimeWarning: invalid value encountered in less
    return (self.a &lt; x) &amp; (x &lt; self.b)
  C:\Users\Jeremy\Anaconda3\lib\site-packages\scipy\stats\_distn_infrastructure.py:1818: RuntimeWarning: invalid value encountered in less_equal
    cond2 = cond0 &amp; (x &lt;= self.a)
  </code></pre>
  
  <p>A snippet of the actual dataframe:</p>
  
  <pre><code>no  NED z   obj_type    S_21    power   SI_flag U_mag   B_mag   V_mag   R_mag   K_mag   W1_mag  W2_mag  W3_mag  W4_mag  L_UV    Q   flag_uv
  1   SDSSJ000005.95+145310.1 2.499   *   0.0 0.0     -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  0.0 0.0 NONE
  4   SDSSJ000009.27+020621.9 1.432   UvS 0.0 0.0     -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  0.0 0.0 NONE
  5   SDSSJ000009.38+135618.4 2.239   QSO 0.0 0.0     -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  0.0 0.0 NONE
  6   SDSSJ000011.37+150335.7 2.18    *   0.0 0.0     -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  0.0 0.0 NONE
  11  SDSSJ000030.64-064100.0 2.606   QSO 0.0 0.0     -999.0  -999.0  -999.0  -999.0  15.46   -999.0  -999.0  -999.0  -999.0  23.342  56.211000000000006  UV
  15  SDSSJ000033.05+114049.6 0.73    UvS 0.0 0.0     -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  -999.0  0.0 0.0 NONE
  27  LBQS2358+0038   0.95    QSO 0.0 0.0     17.342  18.483  18.203  17.825  -999.0  -999.0  -999.0  -999.0  -999.0  23.301  56.571999999999996  UV
  </code></pre>
  
  <p>I'm plotting each of the <code>_mag</code> columns against <code>z</code>, and I'm trying to compute and plot a linear regression, excluding the <code>NaN</code>s.</p>
  
  <p>I've tried <code>numpy.linalg</code>, <code>numpy.poly</code>, <code>scipy.stats.linregress</code>, and <code>statsmodels.api</code>, but it doesn't seem like any of them can easily handle the <code>NaN</code>s. The other questions I've found on SE are leading me around in circles.</p>
  
  <p>How can I plot the OLS regression fit over the top of my data, like the MWE shows?</p>