title: XML to Postgres via python/psycopg2
body: <p>I have an existing python script that loops through a directory of XML files parsing each file using etree, and inserting data at different points into a Postgres database schema using psycopg2 module.  This hacked together script worked just fine but now the amount of data (number and size of XML files) is growing rapidly, and the number of INSERT statements is just not scaling. The largest table in my final database has grown to about ~50 million records from about 200,000 XML files. So my question is, what is the most efficient way to:</p>
  
  <ol>
  <li>Parse data out of XMLs</li>
  <li>Assemble row(s)</li>
  <li>Insert row(s) to Postgres</li>
  </ol>
  
  <p>Would it be faster to write all the data to a CSV in the correct format and then bulk load the final CSV tables to Postgres using COPY_FROM command?  </p>
  
  <p>Otherwise I was thinking about populating some sort of temporary data structure in memory that I could insert into the DB once it reaches a certain size?  I am just having trouble arriving at the specifics of how this would work.  </p>
  
  <p>Thanks for any insight on this topic, and please let me know if more information is needed to answer my question.</p>