title: Scraping text without javascript code using scrapy
body: <p>I'm currently setting up a bunch of spiders using scrapy. These spiders are supposed to extract <strong>only text</strong> (articles, forum posts, paragraphs, etc) from the target sites.</p>
  
  <p>The problem is : sometimes, my target node contains a <code>&lt;script&gt;</code> tag and so the scraped text contains javascript code.</p>
  
  <p><a href="http://www.aufeminin.com/societe/colliers-d-ambre-chez-les-bebes-attention-danger-s6952.html" rel="noreferrer">Here is a link</a> to a real example of what I'm working with. In this case my target node is <code>//td[@id='contenuStory']</code>. The problem is that there's a <code>&lt;script&gt;</code> tag in the first child div.</p>
  
  <p>I've spent a lot of time searching for a solution on the web and on SO, but I couldn't find anything. I hope I haven't missed something obvious !</p>
  
  <h2>Example</h2>
  
  <p>HTML response (only the target node) :</p>
  
  <pre><code>&lt;div id="content"&gt;
      &lt;div id="part1"&gt;Some text&lt;/div&gt;
      &lt;script&gt;var s = 'javascript I don't want';&lt;/script&gt;
      &lt;div id="part2"&gt;Some other text&lt;/div&gt;
  &lt;/div&gt;
  </code></pre>
  
  <p>What I want in my item :</p>
  
  <pre><code>Some text
  Some other text
  </code></pre>
  
  <p>What I get :</p>
  
  <pre><code>Some text
  var s = 'javascript I don't want';
  Some other text
  </code></pre>
  
  <h2>My code</h2>
  
  <p>Given an xpath selector I'm using the following function to extract the text :</p>
  
  <pre><code>def getText(hxs):
      if len(hxs) &gt; 0:
          l = hxs.select('string(.)')
          if len(l) &gt; 0:
              s = l[0].extract().encode('utf-8')
          else:
              s = hxs[0].extract().encode('utf-8')
          return s
      else:
          return 0
  </code></pre>
  
  <p>I've tried using XPath axes (things like <code>child::script</code>) but to no avail.</p>