title: How to perform up-sampling using sample() function(py-spark)
body: <p>I am working on a Binary Classification Machine Learning Problem and I am trying to balance the training set as I have an imbalanced target class variable. I am using Py-Spark for building the model.</p>
  
  <p>Below is the code which is working to balance the data</p>
  
  <pre><code>train_initial, test = new_data.randomSplit([0.7, 0.3], seed = 2018)
  train_initial.groupby('label').count().toPandas()
     label   count                                                                
  0    0.0  712980
  1    1.0    2926
  train_new = train_initial.sampleBy('label', fractions={0: 2926./712980, 1: 1.0}).cache()
  </code></pre>
  
  <p>The above code performs under-sampling, but I think this might lead to loss of information. However, I am not sure how to perform upsampling. I also tried to use sample function as below:</p>
  
  <pre><code>train_up = train_initial.sample(True, 10.0, seed = 2018)
  </code></pre>
  
  <p>Although, it increases the count of 1 in my data set, it also increases the count of 0 and gives the below result.</p>
  
  <pre><code>   label    count                                                               
  0    0.0  7128722
  1    1.0    29024
  </code></pre>
  
  <p>Can someone please help me to achieve up-sampling in py-spark.</p>
  
  <p>Thanks a lot in Advance!!   </p>