title: Why does performance decrease with the size of the data frame?
body: <p>I'm working with a relatively large dataset (approx 5m observations, made up of about 5.5k firms).</p>
  
  <p>I needed to run OLS regressions with a 60 month rolling window for each firm. I noticed that the performance was insanely slow when I ran the following code:</p>
  
  <pre><code>for idx, sub_df in master_df.groupby("firm_id"):
      # OLS code
  </code></pre>
  
  <p>However, when I first split my dataframe into about 5.5k dfs and then iterated over each of the dfs, the performance improved dramatically.</p>
  
  <pre><code>grouped_df = master_df.groupby("firm_id")
  df_list = [group for group in grouped_df]
  
  for df in df_list:
      my_df = df[1]
      # OLS code
  </code></pre>
  
  <p>I'm talking 1-2 weeks of time (24/7) to complete in the first version compared to 8-9 hours tops.</p>
  
  <p>Can anyone please explain <strong>why</strong> splitting the master df into N smaller dfs and then iterating over each smaller df performs better than iterating over the same number of groups within the master df?</p>
  
  <p>Thanks ever so much!</p>