title: Memory Leak Merging 2 Pandas Dataframe
body: <p>I am not the first one asking about this issue however my problem seems a bit different from what I saw. Basically I was trying to merge a dataframe <code>A.shape = [900k, 6]</code> and <code>B.shape = [600k, 6]</code> and I get a memory error with a <code>pd.merge(A, B, how='left', left_on=[col_1, col_2], right_on=[col_1, col_2])</code>. (I have 5GB of RAM available for the computation.)</p>
  
  <p>To try to circunvent it, I tried to do it iteratively like below but after the 6 first iterations where the memory computation time increases slightly from one to the other, it explodes at the 7th one:</p>
  
  <pre><code>def partial_merge(A, partial_B):
      return pd.merge(A, partial_B, 
                   how='left', 
                   left_on=['col_1','col_2'], 
                   right_on = ['col_1','col_2'])
  
  low_bound = 0
  for high_bound in tqdm(range(0, merged_stats.shape[0], 10000)):
      A = partial_merge(A, B.iloc[low_bound:high_bound, :])
      gc.collect()
      low_bound = high_bound
  </code></pre>
  
  <p>Any idea ?
  I am reaching the point where I am going to write those 2 dataframes to a database to do the join there.</p>