title: Read CSV File into Pandas Dataframe with Chunking Resulting in a Single Target Dataframe
body: <p>I need to read a large 4GB file as csv in Pandas into a dataframe. Here is my current code:</p>
  <pre><code>source_data_df = pd.read_csv(temp_file, skipinitialspace=True, dtype=str, na_values=['N.A.'])
  </code></pre>
  <p>It fails due to out of memory error.</p>
  <p>I know I can use chunksize Pandas option to reduce memory utilization and process data in chunks before saving to disk. I can't figure out how to create the resulting single dataframe with all the data from csv file from the chunks before saving it to disk. I need help finishing this code to create this final dataframe. Could someone help?</p>
  <pre><code>chunksize = 10000000
  with pd.read_csv(temp_file, skipinitialspace=True, dtype=str, na_values=['N.A.'], chunksize=chunksize) as reader:
      for chunk in reader:
          process(chunk)
  </code></pre>
  <p>Alternatively, can the processed chunks be appended to a single target file on disk? How would the above code look then?</p>
  <p>Also, need a recommendation on the optimal chunksize for 3 GB file, does chunksize represent the size of a chunk in bytes? What would it be for say 500 MB chunks?</p>
  <p>What units the chunksize is measured in? I assumed it is bytes, can someone confirm?</p>
  <p>Thank you very much!</p>