title: Python LXML iterparse function: memory not getting freed while parsing a huge XML
body: <p>I am parsing big XMLs (~500MB) with the help of LXML library in Python. I have used BeautifulSoup with lxml-xml parser for small files. But when I came across huge XMLs, it was inefficient as it reads the whole file once, and then parses it.  </p>
  
  <p>I need to parse a XML to get root to leaf paths (except the outermost tag).<br>
  eg.</p>
  
  <pre><code>&lt;?xml version="1.0" encoding="utf-8"?&gt;
  &lt;!DOCTYPE A&gt;
  &lt;A&gt;
      &lt;B&gt;
          &lt;C&gt;
              abc
          &lt;/C&gt;
          &lt;D&gt;
              abd
          &lt;/D&gt;
      &lt;/B&gt;
  &lt;/A&gt;
  </code></pre>
  
  <p>Above XML should give keys and values as output (root to leaf paths).</p>
  
  <p>A.B.C = abc<br>
   A.B.D = abd</p>
  
  <p>Here's the code that I've written to parse it:<br>
  (ignore1 and ignore2 are the tags that need to be ignored, and tu.clean_text() is the function which will remove unnecessary characters) </p>
  
  <pre><code>def fast_parser(filename, keys, values, ignore1, ignore2):
      context = etree.iterparse(filename, events=('start', 'end',))
  
      path = list()
      i = 0
      lastevent = ""
      for event, elem in context:
          i += 1
          tag = elem.tag if "}" not in elem.tag else elem.tag.split('}', 1)[1]
  
          if tag == ignore1 or tag == ignore2:
              pass
          elif event == "start":
              path.append(tag)
          elif event == "end":
              if lastevent == "start":
                  keys.append(".".join(path))
                  values.append(tu.clean_text(elem.text))
  
              # free memory
              elem.clear()
              while elem.getprevious() is not None:
                  del elem.getparent()[0]
              if len(path) &gt; 0:
                  path.pop()
          lastevent = event
  
      del context
      return keys, values
  </code></pre>
  
  <p>I have already referred the following article for parsing a large file <a href="http://ibm.com/developerworks/xml/library/x-hiperfparse/#listing4" rel="nofollow noreferrer">ibm.com/developerworks/xml/library/x-hiperfparse/#listing4</a> </p>
  
  <p>Here's the screenshot of top command. Memory usage goes beyond 2 GB for a ~500 MB XML file. I suspect that memory is not getting freed.
  <a href="https://i.stack.imgur.com/bDj6Q.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bDj6Q.png" alt="enter image description here"></a></p>
  
  <p>I have already gone through few StackOverflow questions. But it didn't help. Please advice.</p>