title: pandas: optimizing my code (groupby() / apply())
body: <p>I have a dataframe of shape (RxC) 1.5M x 128. I do the following:</p>
  
  <ol>
  <li>I do groupby() based on 6 columns. This creates ~8700 sub-groups each of shape 538 x 122.</li>
  <li>On each sub-group, I run apply(). This function computes the % frequency of each categorical value PER column (i.e., 122) in the sub-group.</li>
  </ol>
  
  <p>So my (pesudo) code:</p>
  
  <p><code>&lt;df = Read dataframe from file&gt;
  g = df.groupby(grp_cols)
  g[nongrp_cols].apply(lambda d: d.apply(lambda s: s.value_counts()) / len(d.index))</code></p>
  
  <p>The code is working OK for me so now I'm profiling it to improve performance. The apply() function takes about 20-25 minutes to run. I believe the problem is it is iterating over every column (122 times) for 8700 times (each subgroup) which may not be the best way (given the way I have coded it). </p>
  
  <p>Can anyone recommend ways I can try to speed this up? </p>
  
  <p>I tried using python multiprocessing pool  (8 processes) to divide the subgroups into equal sets to process, but ended up getting some pickling error...</p>
  
  <p>Thanks.</p>