title: Skipping perfectly identical records in SQL Server
body: <p>I have a set of text files I'm importing into SQL Server via a SqBulkCopy in C#.  There are some records that appear in multiple records and I'd like to skip those rather than throwing an error and stopping.</p>
  
  <p>For cases where there's a clear 'uniqueID'/primary key, all is well - I've used the "WITH (IGNORE_DUP_KEY = ON)" with good effect.</p>
  
  <p>In this case though, no such uniqueID can be determined, short of joining ALL the fields together, or creating a compound key that involves all the fields.  I've tried to get a unique transactionID or similar but none is available.  Sometimes the records are the same but one digit in a field has changed; in this case I want it inserting into the table.  </p>
  
  <p>I figure there may be a better approach than creating a key against all fields?  If I did do that, it would mean dates, strings, values etc would all have to be mixed in to form the key.  </p>
  
  <p>Querying the rows out and hashing the values for comparison with a hash of my input line was another thought, but performance would be terrible I'm sure :)</p>
  
  <p>Anyone have suggestions on how best to do this?</p>
  
  <p>Thanks</p>