title: How to split large spark dataframe(5m rows)/csv file into chunk with same header for each dataframe(Optimized solution)
body: <p>I have large csv file(5GB) which contains the ~50M rows. I want to make the smaller chunk csv(~1M rows) from original csv file. I tried below approaches to chunk but takes lots of time to perform the chunk:-</p>
  <pre><code>1. I used Pandas to read data from csv and make chunk and write into chunk csv file.
  
  import pandas
  rows = pd.read_csv('test.csv', chunksize=1000000)
  for i, chunck in enumerate(rows):
       chunck.to_csv('/out{}_{}.csv'.format(i, 'test'))
  For this execution it takes around 3 mins.
  
  2. I used DASK to perform the same operation
  
  import dask.dataframe
  data = dask.dataframe.read_csv('test.csv')
  data.to_csv('/test-*.csv')
  For this it takes the same time 3 mins
  </code></pre>
  <p>SO, I tried to use spark dataframe to reduce the execution time. I am able to read csv successfully from pyspark but not able to make chunks(dataframes) with the same header for each chunk so, I can write the each chunk into individual csv file.</p>
  <p>Could you help me how to make smallest chunks with same header and write into csv using pyspark?</p>