title: cleaning data with dropna in Pyspark
body: <p>I'm still relatively new to Pyspark. I use version 2.1.0.
  I'm trying to clean some data on a much larger data set.
  I've successfully used several techniques such as "dropDuplicates" along with subsets and sql functions (distinct, count etc).</p>
  
  <p>I then came across dropna, which I thought might simplify matters. But I do not understand why rows 3 and 6 are still present after using dropna.
  For example:</p>
  
  <pre><code>df = spark.createDataFrame([(1, 'Peter', 1.79, 28,'M', 'Tiler'),
                              (2, 'Fritz', 1.78, 45,'M', None),
                              (3, 'Florence', 1.75, None, None, None),
                              (4, 'Nicola',1.6, 33,'F', 'Dancer'),
                              (5, 'Gregory', 1.8, 54,'M', 'Teacher'),
                              (6, 'Steven', 1.82, None, 'M', None),
                              (7, 'Dagmar', 1.7, 42,'F', 'Nurse'),]
                             , ['id', 'Name', 'Height', 'Age', 'Gender', 'Occupation'])
  
  df.show()
  
  df.dropna(thresh=2)
  
  df.show()
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>+---+--------+------+----+------+----------+
  | id|    Name|Height| Age|Gender|Occupation|
  +---+--------+------+----+------+----------+
  |  1|   Peter|  1.79|  28|     M|     Tiler|
  |  2|   Fritz|  1.78|  45|     M|      null|
  |  3|Florence|  1.75|null|  null|      null|
  |  4|  Nicola|   1.6|  33|     F|    Dancer|
  |  5| Gregory|   1.8|  54|     M|   Teacher|
  |  6|  Steven|  1.82|null|     M|      null|
  |  7|  Dagmar|   1.7|  42|     F|     Nurse|
  +---+--------+------+----+------+----------+
  
  +---+--------+------+----+------+----------+
  | id|    Name|Height| Age|Gender|Occupation|
  +---+--------+------+----+------+----------+
  |  1|   Peter|  1.79|  28|     M|     Tiler|
  |  2|   Fritz|  1.78|  45|     M|      null|
  |  3|Florence|  1.75|null|  null|      null|
  |  4|  Nicola|   1.6|  33|     F|    Dancer|
  |  5| Gregory|   1.8|  54|     M|   Teacher|
  |  6|  Steven|  1.82|null|     M|      null|
  |  7|  Dagmar|   1.7|  42|     F|     Nurse|
  +---+--------+------+----+------+----------+
  </code></pre>
  
  <p>Could somebody suggest why the rows aren't removed?</p>
  
  <p>The pyspark <a href="https://github.com/apache/spark/blob/e2773996b8d1c0214d9ffac634a059b4923caf7b/python/pyspark/sql/tests.py" rel="noreferrer">examples</a> show correct counts as per the usage I assumed.</p>
  
  <pre><code># threshold
          self.assertEqual(self.spark.createDataFrame(
              [(u'Alice', None, 80.1)], schema).dropna(thresh=2).count(),
              1)
          self.assertEqual(self.spark.createDataFrame(
              [(u'Alice', None, None)], schema).dropna(thresh=2).count(),
              0)
  </code></pre>