Sign up ×

I have read several options (PERL, Ruby, Python and PHP) about web scraping a website or RSS feed. My question is, if I used say Python or Ruby to web scrape, is it possible to write the scrape to an .html file like the index.html? Basically what I wanna do is grab x,y,z links from an RSS feed and make a menu on my home page out of x,y,z.

Is this possible? If so, where would I get started?

@Amigable Clark Kant

Firstly, thanks for the more detailed answer. So each time I scrape, I'd have to rewrite the whole HTML file? I have one section, Headlines, that I need to change every hour or so, which gets the information from the scrape. Here is the code that I would need to be changed every time the scrape runs:

<article class="grid_4">
        <div class="box-1">
          <ul class="list-1 p2">
            <li><a href="more.html"><b></b><strong>Lorem ipsum</strong> dolor amet, consectetuer</a></li>
            <li><a href="more.html"><b></b><strong>Adipiscing elit</strong> sed diam nonummy nibh</a></li>
            <li><a href="more.html"><b></b><strong>Euismod tincidunt</strong> laoreet dolore magna</a></li>
            <li><a href="more.html"><b></b><strong>Aliquam erat</strong> volutpat wisi enim ad minim</a></li>
            <li><a href="more.html"><b></b><strong>Veniam nostrud</strong> exerci tation ullamcorper</a></li>
            <li><a href="more.html"><b></b><strong>Suscipit lobortis</strong> nisl ut aliquip commodo</a></li>
            <li><a href="more.html"><b></b><strong>Duis autem</strong> vel eum iriure dolor hendrerit</a></li>
            <li><a href="more.html"><b></b><strong>In vulputate</strong> velit esse molestie consequat</a></li>
            <li><a href="more.html"><b></b><strong>Vel illum</strong> dolore eu feugiat nulla facilisis</a></li>
            <li><a href="more.html"><b></b><strong>At vero</strong> eros et accumsan</a></li>
          <div class="alignright p3"><a href="more.html" class="link-1">read more</a></div>
        <a href="more.html" class="banner-1"></a> </article>

Is it possible to do this? If not, can you point me somewhere to start reading/learning about it? Thanks again.

share|improve this question

closed as not constructive by katrielalex, Lev Levitsky, Soner Gönül, Andy Hayden, Graviton Jan 21 '13 at 4:48


You can scrape/parse anything: a web request is a web request, the response is the response. Search the web for any tutorial on scraping in whichever language you want. –  Dave Newton Jan 15 '13 at 15:36
@ esaelPsnoroMoN - I'm not asking what to use to parse, I already know that and I know that it has been asked 10000 times. What I'm asking is, once you have the data you want, how do I get it to the actual HTML page. –  Rodney Wilson Jan 15 '13 at 17:52

2 Answers 2

Install Beautiful Soup with

pip install beautifulsoup4

then read the documentation

( If you don't have pip, install it from here: )

Beautiful Soup will give you a nice clean version of the web page. You can search through this structure looking for links or whatever information you are after.


Will give you all the links in a page.

To get these onto a web page there is an infinite number of possibilities, some come to mind:

  • You could generate the web page

  • You could generate a web page and load it in an iframe

  • You could generate a text file in JSON format with all the links in it and GET it from Javascript embedded on a static web page

  • You could run Django, as you say

What is best depends on a lot of factors - how often can you scrape? How often should you scrape? Does the scraping have to happen on the web server or should it happen on some other computer which periodically uploads to your web server?

Fuzzy question, fuzzy answer...

share|improve this answer
I know how to use BeautifulSoup in Python. I'm asking once I have the data I want, how do I get it to my HTML page. Would I have to use DJango or some other template software? –  Rodney Wilson Jan 15 '13 at 17:53

yes it is possible...

python/beautiful soup

more soup


share|improve this answer
