title: How to check specific partition data from Spark partitions in Pyspark
body: <p>I have a created two dataframes in pyspark from my hive table as:</p>
  
  <pre><code>data1 = spark.sql("""
     SELECT ID, MODEL_NUMBER, MODEL_YEAR ,COUNTRY_CODE
     from MODEL_TABLE1 where COUNTRY_CODE in ('IND','CHN','USA','RUS','AUS')
  """);
  </code></pre>
  
  <p>each country is having millions of unique ID in alphanumeric format.</p>
  
  <pre><code>data2 = spark.sql("""
     SELECT ID,MODEL_NUMBER, MODEL_YEAR, COUNTRY_CODE
     from MODEL_TABLE2 where COUNTRY_CODE in ('IND','CHN')
  """);
  </code></pre>
  
  <p>I want to join both of these dataframe using pyspark on ID column.</p>
  
  <p>How can we re-partition our data so that its get distributed uniformly across the partitions.</p>
  
  <p>Can i use below to reparation my data?</p>
  
  <pre><code>newdf1 = data2.repartition(100, "ID")
  newdf2 = data2.repartition(100, "ID")
  </code></pre>
  
  <p>what would be the best way for partitioning so that join work faster?</p>