title: How do I convert a multiprocessor.manager.list to a pure python list
body: <p>I have my normal script doing about 30,000 records in 20 seconds. Given that amount of data that I have to run through (over 50 million records), I thought it wise to use python's multiprocessing.</p>
  
  <p>At the end of my process, I do a database update using sqlalchemy core where I update the processed records in batches of 50,000. <a href="http://docs.sqlalchemy.org/en/rel_0_9/core/tutorial.html#executing-multiple-statements" rel="noreferrer">SQLAlchemy Core requires that you pass it a list for it to do a bulk update or even insert</a>. I'll call this list <code>py_list</code></p>
  
  <p>For Python's multiprocessing, am capturing the results of the processes via a <code>multiprocessing.manager.list()</code> which I will call <code>mp_list</code>. </p>
  
  <p>Everything works fine till the point that I pass the <code>mp_list</code> to the SQLAlchemy bulk update statement. This fails with the error <code>AttributeError: 'list' object has no attribute 'keys'</code>. Googling brings me to a <a href="https://stackoverflow.com/a/10807925/1082673">question on SO</a> which states that the multiprocessing.manager.list() and even a multiprocessing.manager.dict() is/are not a true python lists/dictionaries.</p>
  
  <p>Question then is, how do I convert the multiprocessing.manager.list into a true python list.</p>
  
  <p><code>mp_list</code> is populated as follows:</p>
  
  <pre><code>import multiprocessing
  manager = multiprocessing.Manager()
  mp_list = manager.list()
  
  def populate_mp_list(pid, is_processed):
      '''Mark the record as having been processed'''
      dict = {}
      dict['b_id'] = pid
      dict['is_processed'] = is_processed
      mp_list.append(dict)
  </code></pre>
  
  <p>The SQLALchemy code throwing the error is as follows:</p>
  
  <pre><code>CONN = Engine.connect()
  trans = CONN.begin()
  stmt = mytable.update().where(mytable.c.id == bindparam('b_id')).\
  values(is_processed=bindparam('is_processed'))
  CONN.execute(stmt, mp_list)
  trans.commit(
  </code></pre>
  
  <p>I've tried converting the mp_list into a true python list. The new list created works but the time penalty for its creation negates all the time saved in multiprocessing.</p>
  
  <p>If I do a loop of the returned <code>mp_list</code> and create a new list.</p>
  
  <pre><code>y = []
  for x in mp_list:
      y.append(x)
  </code></pre>
  
  <p>Also, if I do a "copy" of the <code>mp_list</code>, each copy adds a 3 seconds! penalty on average which ain't cool.</p>
  
  <pre><code>y = mp_list[0:len(mp_list)]
  </code></pre>
  
  <p>So, which would be the fastest way to convert the multiprocessing.manager.list into a list usable by SQLAlchemy Core?</p>