title: What algorithm to use to delete duplicates?
body: <p>Imagine that we have some file, called, for example, "A.txt". We know that there <strong>are</strong> some duplicate elements. "A.txt" is <strong>very</strong> big, more than ten times bigger than memory, maybe around 50GB. Sometimes, size of B will be approximately equal to size of A, sometimes it will be <em>many</em> times smaller than size of A.
  Let it have structure like that:</p>
  
  <pre><code>a 1
  b 2
  c 445
  a 1
  </code></pre>
  
  <p>We need to get file "B.txt", that will not have such duplicates. As example, it should be this:</p>
  
  <pre><code>a 1
  b 2
  c 445
  </code></pre>
  
  <p>I thought about algorithm that copy A and does B, then takes first string in B, and look for each another, if finds the same, deletes duplicates. Then takes second string, etc.</p>
  
  <p>But I think it is <strong>way</strong> too slow. What can I use?</p>
  
  <p><em>A is <strong>not</strong> database! No SQL, please.</em></p>
  
  <p>Sorry, that didn't said, sorting is OK.</p>
  
  <p><em>Although it can be sorted, what about if it cannot be sorted?</em></p>