title: set some rule to groupby in pandas
body: <p>I need to set some rule to groupby in pandas. I hope I can ignore the rows if ['keep'] column have &quot;dup by&quot; before I groupby the datetime.</p>
  <p>There is my code:</p>
  <pre><code>import pandas as pd
  import numpy as np
  
  df = pd.read_csv(&quot;sample.csv&quot;,delimiter='|')
  
  df['datetime'] = pd.to_datetime(df['datetime'],errors = 'coerce')
  most_recent_date = df.groupby(df['VIP_ID'])['datetime'].max()
  most_recent_date= most_recent_date.rename(&quot;most_recent_date&quot;)
  df = df.join(most_recent_date, on=&quot;VIP_ID&quot;)
  
  df['both'] = np.where(
         ((df['keep'] == 'same tier')&amp;(dup == 'yes')),
            df['VIP_ID']+df['datetime'].astype(str),
           df['ID']
  )
  df['keep'] = np.where(
      df['keep'] != 'same tier',df['keep'],
      (np.where(
           df['most_recent_date'] == df['datetime'],
           'yes',
           'dup by ' + df['VIP_ID'].astype(str)))
  )
  
  
  
  df.loc[df.duplicated(subset=['both'], keep = False),'keep'] = 'same time'
  df = df.drop(columns = ['both','most_recent_date'])
  print(df)
  </code></pre>
  <p>And this code make all keep column become 'dup by'.</p>
  <p>example csv:</p>
  <pre><code>ID|VIP_ID|TIER|datatime|keep
  1|F08210020403|GO|2014-05-17 00:00:00|same tier
  2|F08210020403|GO|2014-04-18 00:00:00|same tier
  3|F08210020403|FO||dup by F08210020403
  4|F08210020403|FO||dup by F08210020403
  5|F08210020403|FO|2016-09-18 00:00:00|dup by F08210020403
  6|F08210020403|FO|2016-05-10 00:00:00|dup by F08210020403
  7|F08210020403|FO||dup by F08210020403
  8|F08210020403|FO||dup by F08210020403
  </code></pre>
  <p>Because 2016-05-10 00:00:00 is the max datetime by F08210020403, all keep columns will show dup by F08210020403.I hope I can set some rules about if keep contain 'dup', ignore this row. After than
  groupby remain rows.</p>
  <p>This is my output:</p>
  <pre><code>1|F08210020403|GO|2014-05-17 00:00:00|dup by F08210020403
  2|F08210020403|GO|2014-04-18 00:00:00|dup by F08210020403
  3|F08210020403|FO||dup by F08210020403
  4|F08210020403|FO||dup by F08210020403
  5|F08210020403|FO|2016-09-18 00:00:00|dup by F08210020403
  6|F08210020403|FO|2016-05-10 00:00:00|dup by F08210020403
  7|F08210020403|FO||dup by F08210020403
  8|F08210020403|FO||dup by F08210020403
  </code></pre>
  <p>expect output:</p>
  <pre><code>1|F08210020403|GO|2014-05-17 00:00:00|yes
  2|F08210020403|GO|2014-04-18 00:00:00|dup by F08210020403
  3|F08210020403|FO||dup by F08210020403
  4|F08210020403|FO||dup by F08210020403
  5|F08210020403|FO|2016-09-18 00:00:00|dup by F08210020403
  6|F08210020403|FO|2016-05-10 00:00:00|dup by F08210020403
  7|F08210020403|FO||dup by F08210020403
  8|F08210020403|FO||dup by F08210020403
  </code></pre>
  <p>Any help would be very much appreciated.</p>