title: Find and remove duplicates in a CSV file
body: <p>I have a large CSV file (1.8 GB) with three columns. Each row contains two strings and a numerical value. The problem is that they are duplicate but swapped.
  Example:</p>
  
  <pre><code>Col1,Col2,Col3
  ABC,DEF,123
  ABC,EFG,454
  DEF,ABC,123
  </code></pre>
  
  <p>The desired output would look like this:</p>
  
  <pre><code>Col1,Col2,Col3
  ABC,DEF,123
  ABC,EFG,454
  </code></pre>
  
  <p>because the third row contains the same information like the first row.</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>The data basically looks like this (Strings in the first two columns and a numerical value in the third, 40 Million lines):</p>
  
  <blockquote>
    <p><a href="https://i.stack.imgur.com/fcBTa.png" rel="nofollow noreferrer">Blockquote</a></p>
  </blockquote>