title: Implementing multiprocessing in a loop scraper and appending the data
body: <p>I am making a web scraper to build a database. The site I plan to use has index pages each containing 50 links. The amount of pages to be parsed is estimated to be around 60K and up, this is why I want to implement multiprocessing.</p>
  
  <p>Here is some pseudo-code of what I want to do:</p>
  
  <pre><code>def harvester(index):
      main=dict()
      ....
      links = foo.findAll ( 'a')
      for link in links:
          main.append(worker(link))
          # or maybe something like: map_async(worker(link))
  def worker(url):
      ''' this function gather the data from the given url'''
      return dictionary
  </code></pre>
  
  <p>Now what I want to do with that is to have a certain number of worker function to gather data in parallel on different pages. This data would then be appended to a big dictionary located in harvester or written directly in a csv file by the worker function.</p>
  
  <ul>
  <li>I'm wondering how I can implement parallelism. I have done a faire
  amount of research on using gevent, threading and multiprocessing but
  I am not sure how to implement it.</li>
  <li>I am also not sure if appending data to a large dictionary or writing
  directly in a csv using DictWriter will be stable with that many input at the same time.</li>
  </ul>
  
  <p>Thanks</p>