title: Pandas Dataframe duplicate records picking up incorrect records
body: <p>I am using Pandas, Jupyter Notebooks and Python. I have a dataset with with 4 columns and 10000 records. Currently when I use the following code to pick up duplicates, somehow the code is picking up incorrect records. FYI: the datatype of the columns are as follow:</p>
  
  <pre><code>Initial_Date = int64
  Final_Date = int64
  Origin = object
  sub_location = object
  </code></pre>
  
  <p>My current code is:</p>
  
  <pre><code>dup = df.duplicated(['Initial_Date','Final_Date','Origin','sub_location'], keep='last')
  </code></pre>
  
  <p>Here is an example of the dataset that is being picked up using the above code:</p>
  
  <pre><code>00121980,00121980,Australia,Brighton:Queensland
  00121980,00121980,Australia,Brisbane:Queensland
  17021987,17021987,Bangladesh,Sylhet-Sunamganj
  17021987,17021987,Brazil,Sao Paolo suburb
  </code></pre>
  
  <p>If you look at the first two records: the initial and final date and Origin are matching however the sub_origin is not matching, one is Brighton and the other is Brisbane.</p>
  
  <p>Same applies in the last two records, the dates are matching but Origin is not the same.</p>
  
  <p>From this, I understand that df.duplicated is not picking up correct records or I am not using it properly. Do data types matter with df.duplicated?</p>
  
  <p>If I just use <code>df.duplicated</code> then the boolean series that is returned has NO duplicates. Can someone please explain/show me how .duplicated is used?</p>
  
  <p>Please keep in mind that this is not the full dataset however the example that I have presented is exactly the problem I have in the real dataset. I narrowed the df.duplicated criteria and came across this error.</p>
  
  <p>Thanks guys :D</p>