title: Q: PySpark, how to iterate over rows in a large datafram?
body: <p>I have a large data frame containing at least half a million records, I need to convert each string to a custom object(should end up with a list of objects)
  collect () is not acceptable because the data frame is too large.</p>
  <p>There was a problem with iterating over all the rows in the data frame, at first I tried to do it like this (led to a map, and a map to a list of objects):</p>
  <pre><code>result_list = map(Lambda row: row, test_df.collect())
  </code></pre>
  <p>But collect () turned out to be unacceptable because the dataframe is too big.</p>
  <p>Could you please tell me what other options are there to iterate over each row in the dataframe?</p>