title: how to write content of df into csv file using multiprocessing in python
body: <p>I have a function that writes the content of df into csv file. </p>
  
  <pre><code>def writeToCSV(outDf, defFile, toFile, retainFlag=True, delim='\t', quotechar='"'):
      headers = []
      fid = open(defFile, 'r')
      for line in fid:
          headers.append(line.replace('\r','').split('\n')[0].split('\t')[0])
      df = pd.DataFrame([], columns=headers)
      for header in outDf.columns.values:
          if header in headers:
              df[header] = outDf[header]
  
      df.to_csv(toFile, sep=delim, quotechar=quotechar, index=False, encoding='utf-8')
  </code></pre>
  
  <p>How can i parallelize this process? Currently i am using following code</p>
  
  <pre><code>def writeToSchemaParallel(outDf, defFile, toFile, retainFlag=True, delim='\t', quotechar='"'):
      logInfo('Start writingtoSchema in parallel...', 'track')
      headers = []
      fid = open(defFile, 'r')
      for line in fid:
          headers.append(line.replace('\r','').split('\n')[0].split('\t')[0])
      df = pd.DataFrame([], columns=headers)
      for header in outDf.columns.values:
          if header in headers:
              df[header] = outDf[header]
      out_Names = Queue()
      cores = min([int(multiprocessing.cpu_count() / 2), int(len(outDf) / 200000)+1])
      #cores=4
      logInfo(str(cores) + 'cores are used...', 'track')
      # split the data for parallel computation
      outDf = splitDf(df, cores)
      # process the chunks in parallel
      logInfo('splitdf called are df divided...', 'track')
      Filenames=[]
      procs = []
      fname=toFile.split("_Opera_output")[0]
      for i in range(0, cores):
          filename=fname+"_"+str(i)+".tsv"
          proc = Process(target=writeToSchema, args=(outDf[i], defFile,filename, retainFlag,delim, quotechar,i))
          procs.append(proc)
          proc.start()
          print 'processing '+str(i)
          Filenames.append(filename)
  # combine all returned chunks
  #   outDf = out_Names.get()
  #   for i in range(1, cores):
  #       outDf = outDf.append(out_q.get(), ignore_index=True)
      for proc in procs:
          proc.join()
      logInfo('Now we merge files...', 'track')
      print Filenames
      with open(toFile,'w') as outfile:
          for fname in Filenames:
              with open(fname) as infile:
                   for line in infile:
                       outfile.write(line)
  </code></pre>
  
  <p>But it didn't work and gives following error</p>
  
  <pre><code>2017-12-17 16:02:55,078 - track - ERROR: Traceback (most recent call last):
  2017-12-17 16:02:55,078 - track - ERROR:   File "C:/Users/sudhir.tiwari/Document
  s/AMEX2/Workspace/Backup/Trunk/code/runMapping.py", line 257, in &lt;module&gt;
  2017-12-17 16:02:55,089 - track - ERROR: writeToSchemaParallel(outDf, defFile, t
  oFile, retainFlag, delim='\t', quotechar='"')
  2017-12-17 16:02:55,153 - track - ERROR:   File "C:\Users\sudhir.tiwari\Document
  s\AMEX2\Workspace\Backup\Trunk\code\utils.py", line 510, in writeToSchemaParalle
  l
  2017-12-17 16:02:55,163 - track - ERROR: with open(fname) as infile:
  2017-12-17 16:02:55,198 - track - ERROR: IOError
  2017-12-17 16:02:55,233 - track - ERROR: :
  2017-12-17 16:02:55,233 - track - ERROR: [Errno 2] No such file or directory: 'C
  :/Users/sudhir.tiwari/Documents/AMEX2/Workspace/Input/work/Schindler_20171130/Sc
  hindler_20171130_0.tsv'
  </code></pre>
  
  <p>And it didn't write into files as when i search the location no files found. I am using multiprocessing to write dataframe into multiple file and then merging all. Split df divide the dataframe into n parts.</p>