title: Correlation matrix of two Pandas dataframe, with P values
body: <p>I was using this function (see bottom) to calculate both Pearson and Pval starting from two dataframes, but I am not confident with Pval results: it seems that too many negative correlations are significant.</p>
  
  <p>Is there a more elegant way (like one-line-code), in order to calculate Pval along with Pearson?</p>
  
  <p>These two answers (<a href="https://stackoverflow.com/questions/38422001/pandas-dataframe-corrwith-method">pandas.DataFrame corrwith() method</a>) and (<a href="https://stackoverflow.com/questions/38422893/correlation-matrix-of-one-dataframe-with-another">correlation matrix of one dataframe with another</a>) provided elegant solutions, but P values calculation is missing.</p>
  
  <p>Here is the code:</p>
  
  <pre><code>def pearson_cross_map(df1, df2):
      """Correlate each Mvar with each Nvar.
  
      Parameters
      ----------
      df1 : dataframe1
      Shape Mobs X Mvar.
  
      df2 : dataframe2
      Shape Nobs X Nvar.
  
      Returns
      -------
      DFcorr, dataframe Mvar x Nvar in which each element is a Pearson 
  correlation coefficient.
      DFpval, dataframe Mvar x Nvar in which each element is a P value (one-tailed).
  
      """
  
      intersection = (df1.index &amp; df2.index).tolist()
      df1 = df1.convert_objects(convert_numeric=True) 
      df1 = df1.T[intersection].T 
      df1 = df1.loc[:, (df1 != 0).any(axis=0)].sort().sort(axis=1)    
      df2 = df2.convert_objects(convert_numeric=True)
      df2 = df2.T[intersection].T
      df2 = df2.loc[:, (df2 != 0).any(axis=0)].sort().sort(axis=1)
      x = df1.T.values
      y = df2.T.values
      mu_x = x.mean(1)
      mu_y = y.mean(1)
      n = x.shape[1]
      s_x = x.std(1, ddof=n - 1)
      s_y = y.std(1, ddof=n - 1)
      cov = np.dot(x,y.T) - n * np.dot(mu_x[:, np.newaxis], mu_y[np.newaxis, :])
      DFcoeff = pd.DataFrame(cov / np.dot(s_x[:, np.newaxis], s_y[np.newaxis, :]))
      DFcoeff.index = df1.columns.tolist()
      DFcoeff.columns = df2.columns.tolist()
      n = len(intersection)
      r = DFcoeff
      t = r*np.sqrt((n-2)/(1-r*r))
      DFpval = pd.DataFrame(stats.t.cdf(t, n-2))
      DFpval.index = df1.columns.tolist()
      DFpval.columns = df2.columns.tolist()
      return DFcoeff, DFpval
  </code></pre>
  
  <p>Thank you!</p>