title: Get Unique records in Spark
body: <p>I have a <strong>dataframe df</strong> as mentioned below:</p>
  
  <pre><code>**customers**   **product**   **val_id**  **rule_name**  **rule_id** **priority**
       1               A            1           ABC            123         1
       3               Z            r           ERF            789         2
       2               B            X           ABC            123         2
       2               B            X           DEF            456         3
       1               A            1           DEF            456         2      
  </code></pre>
  
  <p>I want to create a new <strong>dataframe df2</strong>, which will have only unique <strong>customer ids</strong>, but as <strong>rule_name</strong> and <strong>rule_id</strong> columns are different for same customer in data, so I want to pick those records which has highest priority for the same <strong>customer</strong>, so my final outcome should be:</p>
  
  <pre><code> **customers**   **product**   **val_id**  **rule_name**  **rule_id** **priority**
           1               A            1           ABC            123         1
           3               Z            r           ERF            789         2
           2               B            X           ABC            123         2
  </code></pre>
  
  <p>Can anyone please help me to achieve it using Spark scala. Any help will be appericiated.</p>