title: lxml.html parsing with XPath and variables
body: <p>I have this HTML snippet</p>
  
  <pre><code>&lt;div id="dw__toc"&gt;
  &lt;h3 class="toggle"&gt;Table of Contents&lt;/h3&gt;
  &lt;div&gt;
  
  &lt;ul class="toc"&gt;
  &lt;li class="level1"&gt;&lt;div class="li"&gt;&lt;a href="#section"&gt;#&lt;/a&gt;&lt;/div&gt;
  &lt;ul class="toc"&gt;
  &lt;li class="level2"&gt;&lt;div class="li"&gt;&lt;a href="#link1"&gt;One&lt;/a&gt;&lt;/div&gt;&lt;/li&gt;
  &lt;li class="level2"&gt;&lt;div class="li"&gt;&lt;a href="#link2"&gt;Two&lt;/a&gt;&lt;/div&gt;&lt;/li&gt;
  &lt;li class="level2"&gt;&lt;div class="li"&gt;&lt;a href="#link3"&gt;Three&lt;/a&gt;&lt;/div&gt;&lt;/li&gt;
  </code></pre>
  
  <p>Now I want to parse it with lxml.html. In the end I want a function where I can provide a searchterm (i.e. "one") and the function should return</p>
  
  <pre><code>One
  #link1
  </code></pre>
  
  <p>For now I'm trying to get a variable in the XPath.</p>
  
  <p><strong>Works:</strong></p>
  
  <pre><code>import lxml.html
  html = lxml.html.parse("www.myurl.com/slash/something")
  
  test=html.xpath("//ul[@class='toc']/li[@class='level2']/div[@class='li']/a/text()='One'")
  
  print test
  </code></pre>
  
  <p>Trying with variable. I want to replace the hardcoded <code>'One'</code> with a variable which I can return to the function later.</p>
  
  <p><strong>Doesn't work:</strong></p>
  
  <pre><code>import lxml.html
  html = lxml.html.parse("www.myurl.com/slash/something")
  
  desiredvars = ['One']
  myresultset=((var, html.xpath("//ul[@class='toc']/li[@class='level2']/div[@class='li']/a[text()='%s']"%(var))[0]) for var in desiredvars)
  
  for each in myresultset: 
          print each
  
  Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;stdin&gt;", line 1, in &lt;genexpr&gt;
  IndexError: list index out of range
  </code></pre>
  
  <p>This is based on this answer: <a href="https://stackoverflow.com/a/10688235/2320453">https://stackoverflow.com/a/10688235/2320453</a>
  Any idea why it doesn't work? Is this the "right way" to do something like this? </p>
  
  <p><strong>EDIT:</strong>
  To sum things up:
  I want to search within the a-Tags and get the text from this Attributes, but I don't want a complete list instead I want to be able to search with a variable.
  Pseudo-code:</p>
  
  <pre><code>import lxml.html
  html = lxml.html.parse("www.myurl.com/slash/something")
  
  searchterm = 'one'
  
  test=html.xpath("...a/text()=searchterm")
  
  print test
  </code></pre>
  
  <p>Expected result</p>
  
  <pre><code>One
  #link1
  </code></pre>