{"action":"created","issue":{"url":"https://api.github.com/repos/stewartmckee/cobweb/issues/53","repository_url":"https://api.github.com/repos/stewartmckee/cobweb","labels_url":"https://api.github.com/repos/stewartmckee/cobweb/issues/53/labels{/name}","comments_url":"https://api.github.com/repos/stewartmckee/cobweb/issues/53/comments","events_url":"https://api.github.com/repos/stewartmckee/cobweb/issues/53/events","html_url":"https://github.com/stewartmckee/cobweb/issues/53","id":220402416,"number":53,"title":"Falling into Crawl Traps","user":{"login":"fuzzygroup","id":4652,"avatar_url":"https://avatars1.githubusercontent.com/u/4652?v=3","gravatar_id":"","url":"https://api.github.com/users/fuzzygroup","html_url":"https://github.com/fuzzygroup","followers_url":"https://api.github.com/users/fuzzygroup/followers","following_url":"https://api.github.com/users/fuzzygroup/following{/other_user}","gists_url":"https://api.github.com/users/fuzzygroup/gists{/gist_id}","starred_url":"https://api.github.com/users/fuzzygroup/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fuzzygroup/subscriptions","organizations_url":"https://api.github.com/users/fuzzygroup/orgs","repos_url":"https://api.github.com/users/fuzzygroup/repos","events_url":"https://api.github.com/users/fuzzygroup/events{/privacy}","received_events_url":"https://api.github.com/users/fuzzygroup/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2017-04-08T14:16:49Z","updated_at":"2017-04-11T08:46:10Z","closed_at":null,"body":"Hi, \r\n\r\nTo start, thank you for an excellent piece of work.  Appreciated.  \r\n\r\nI'm trying to use this to crawl the site http://www.udemy.com/ . I added it to my Gemfile, bundle install and started it up and everything looked great.  What I found was that it fell victim to crawl traps generating urls like this:\r\n\r\nhttps://www.udemy.com/courses/photography/mobile-photography/all-courses/?p=324\r\n\r\nThe actual number pages on the base url: \r\n\r\nhttps://www.udemy.com/courses/photography/mobile-photography/all-courses/\r\n\r\nis only 3 so its very much spidering far, far deeper than needed.\r\n\r\nAny suggestions for how to go about addressing this?  \r\n\r\nWhat I'm trying to do is build a page_archiver and my core loop looks like this (its being executed from a Rake task):\r\n\r\n    statistics = CobwebCrawler.new(:cache => 600, :thread_count => 10, :valid_mime_types => [\"text/html\"]).crawl(\"http://www.udemy.com\") do |page|\r\n      puts \"Just crawled #{page[:url]} and got a status of #{page[:status_code]}.\"\r\n      if page[:mime_type] == \"text/html\"\r\n        page_ctr = page_ctr + 1\r\n        #puts page.title\r\n        #debugger\r\n        page_archive = PageArchive.find_or_create(page[:body], page[:url])\r\n        total_time = (Time.now - start_time) \r\n        puts \"  Total time: #{total_time}\"\r\n        puts \"  Total pages: #{page_ctr}\"\r\n        puts \"  Time per page: #{total_time.to_f / page_ctr}\"\r\n      else\r\n        puts \"  Not text/html for: #{page[:url]}\"\r\n      end\r\n      \r\n    end\r\n\r\nAfter running it for about 20 minutes, it got 10,000 \"pages\" deep almost all of which was just \"psuedo pages\" like the ?p=324 url.\r\n\r\nI didn't see any kind of configuration option that would limit progress so this feels like something internal to the guts of the crawler but if I've missed something, my bad.\r\n\r\nThanks\r\nScott"},"comment":{"url":"https://api.github.com/repos/stewartmckee/cobweb/issues/comments/293192221","html_url":"https://github.com/stewartmckee/cobweb/issues/53#issuecomment-293192221","issue_url":"https://api.github.com/repos/stewartmckee/cobweb/issues/53","id":293192221,"user":{"login":"fuzzygroup","id":4652,"avatar_url":"https://avatars1.githubusercontent.com/u/4652?v=3","gravatar_id":"","url":"https://api.github.com/users/fuzzygroup","html_url":"https://github.com/fuzzygroup","followers_url":"https://api.github.com/users/fuzzygroup/followers","following_url":"https://api.github.com/users/fuzzygroup/following{/other_user}","gists_url":"https://api.github.com/users/fuzzygroup/gists{/gist_id}","starred_url":"https://api.github.com/users/fuzzygroup/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fuzzygroup/subscriptions","organizations_url":"https://api.github.com/users/fuzzygroup/orgs","repos_url":"https://api.github.com/users/fuzzygroup/repos","events_url":"https://api.github.com/users/fuzzygroup/events{/privacy}","received_events_url":"https://api.github.com/users/fuzzygroup/received_events","type":"User","site_admin":false},"created_at":"2017-04-11T08:46:10Z","updated_at":"2017-04-11T08:46:10Z","body":"I think there is properly done forking -- and you're correct about that -- and there is forking where your goal is to fix something for your own needs.  My concern is that this feels to me like a core issue that needs to get addressed by enhancing the core structure and if someone unfamiliar with the code does it (me), they're likely to screw it up badly."}}
{"actor":{"display_login":"fuzzygroup"}}