{"action":"created","comment":{"url":"https://api.github.com/repos/opensyllabus/osp-scraper/pulls/comments/122864759","pull_request_review_id":45015985,"id":122864759,"diff_hunk":"@@ -1,16 +1,85 @@\n import logging\n import fnmatch\n import re\n-from urllib.parse import parse_qs\n+import urllib.parse\n import attr\n-\n+import os\n \n from scrapy import signals\n from scrapy.http import Request\n from scrapy.utils.httpobj import urlparse_cached\n \n logger = logging.getLogger(__name__)\n \n+# a list of base domains to blacklist; subdomains blocked too\n+blacklist_domains = [\n+    'facebook.com',\n+    'reddit.com',\n+    'twitter.com',\n+    'linkedin.com',\n+    'wikipedia.org',\n+]\n+\n+blacklist_re = \"^\" + \"|\".join(\n+    \"((.*\\.)?%s$)\" % re.escape(x)\n+    for x in blacklist_domains\n+)\n+\n+def make_filters(seed_urls):\n+    \"\"\"Generate filters for a spider from a list of URLs.\n+\n+    * Allow paths with matching prefix to infinite depth\n+    * Allow same hostname to max depth of 2\n+    * Allow other domains to max depth of 1\n+\n+    Args:\n+        seed_urls (list)\n+    \"\"\"\n+    if not seed_urls:\n+        raise ValueError(\"List of URLs must be non-empty\")\n+\n+    filters = []\n+\n+    # blacklist domains\n+    filters.append(\n+        Filter.compile(\n+            'deny',\n+            pattern='regex',\n+            hostname=blacklist_re\n+        )\n+    )\n+\n+    # merge parameters from several seed urls, with unique domains & paths\n+    for url in seed_urls:\n+        u = urllib.parse.urlparse(url)\n+        prefix = re.escape(u.path if u.path.endswith('/') else os.path.dirname(u.path) + '/')\n+        hostname=re.escape(u.hostname) if u.hostname is not None else None\n+        port=re.escape(str(u.port)) if u.port is not None else None","path":"osp_scraper/filterware.py","position":59,"original_position":59,"commit_id":"79ffc0d0a5595d5f97f19fc1215172e117e66692","original_commit_id":"79ffc0d0a5595d5f97f19fc1215172e117e66692","user":{"login":"HarrisonGregg","id":3527935,"avatar_url":"https://avatars3.githubusercontent.com/u/3527935?v=3","gravatar_id":"","url":"https://api.github.com/users/HarrisonGregg","html_url":"https://github.com/HarrisonGregg","followers_url":"https://api.github.com/users/HarrisonGregg/followers","following_url":"https://api.github.com/users/HarrisonGregg/following{/other_user}","gists_url":"https://api.github.com/users/HarrisonGregg/gists{/gist_id}","starred_url":"https://api.github.com/users/HarrisonGregg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HarrisonGregg/subscriptions","organizations_url":"https://api.github.com/users/HarrisonGregg/orgs","repos_url":"https://api.github.com/users/HarrisonGregg/repos","events_url":"https://api.github.com/users/HarrisonGregg/events{/privacy}","received_events_url":"https://api.github.com/users/HarrisonGregg/received_events","type":"User","site_admin":false},"body":"Might as well fix the spacing here.","created_at":"2017-06-20T01:52:21Z","updated_at":"2017-06-20T01:52:21Z","html_url":"https://github.com/opensyllabus/osp-scraper/pull/124#discussion_r122864759","pull_request_url":"https://api.github.com/repos/opensyllabus/osp-scraper/pulls/124","_links":{"self":{"href":"https://api.github.com/repos/opensyllabus/osp-scraper/pulls/comments/122864759"},"html":{"href":"https://github.com/opensyllabus/osp-scraper/pull/124#discussion_r122864759"},"pull_request":{"href":"https://api.github.com/repos/opensyllabus/osp-scraper/pulls/124"}}},"pull_request":{"url":"https://api.github.com/repos/opensyllabus/osp-scraper/pulls/124","id":126429718,"html_url":"https://github.com/opensyllabus/osp-scraper/pull/124","diff_url":"https://github.com/opensyllabus/osp-scraper/pull/124.diff","patch_url":"https://github.com/opensyllabus/osp-scraper/pull/124.patch","issue_url":"https://api.github.com/repos/opensyllabus/osp-scraper/issues/124","number":124,"state":"open","locked":false,"title":"Improve crawl parameter creation","user":{"login":"jodizzle","id":672268,"avatar_url":"https://avatars2.githubusercontent.com/u/672268?v=3","gravatar_id":"","url":"https://api.github.com/users/jodizzle","html_url":"https://github.com/jodizzle","followers_url":"https://api.github.com/users/jodizzle/followers","following_url":"https://api.github.com/users/jodizzle/following{/other_user}","gists_url":"https://api.github.com/users/jodizzle/gists{/gist_id}","starred_url":"https://api.github.com/users/jodizzle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jodizzle/subscriptions","organizations_url":"https://api.github.com/users/jodizzle/orgs","repos_url":"https://api.github.com/users/jodizzle/repos","events_url":"https://api.github.com/users/jodizzle/events{/privacy}","received_events_url":"https://api.github.com/users/jodizzle/received_events","type":"User","site_admin":false},"body":"Branched off from #79.\r\n\r\nCurrently, parameters for crawls done by `FilterSpider` are created through invoking the `osp_scraper.tasks.make_params` method inside `edu_repo_crawler.py`.  Three parameters are created and sent to `FilterSpider` as keyword arguments: `start_urls`, `allowed_file_types` and `filters`.  However, creating the `filters` parameter is the only part of this process that is novel: `start_urls` come directly from the spreadsheet and `allowed_file_types` come from a static class variable.  Given this, it seems reasonable to turn `make_params` into `make_filters`, and further, have `FilterSpider` create it's own parameters with a `from_crawler` class method.  This has the additional benefit of making `FilterSpider` easier to subclass.","created_at":"2017-06-20T01:25:25Z","updated_at":"2017-06-20T01:52:21Z","closed_at":null,"merged_at":null,"merge_commit_sha":"0efb1ab285d5ea2a67d12bbbf1d6f04f55f0de38","assignee":null,"assignees":[],"requested_reviewers":[{"login":"wearpants","id":505344,"avatar_url":"https://avatars3.githubusercontent.com/u/505344?v=3","gravatar_id":"","url":"https://api.github.com/users/wearpants","html_url":"https://github.com/wearpants","followers_url":"https://api.github.com/users/wearpants/followers","following_url":"https://api.github.com/users/wearpants/following{/other_user}","gists_url":"https://api.github.com/users/wearpants/gists{/gist_id}","starred_url":"https://api.github.com/users/wearpants/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wearpants/subscriptions","organizations_url":"https://api.github.com/users/wearpants/orgs","repos_url":"https://api.github.com/users/wearpants/repos","events_url":"https://api.github.com/users/wearpants/events{/privacy}","received_events_url":"https://api.github.com/users/wearpants/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/opensyllabus/osp-scraper/pulls/124/commits","review_comments_url":"https://api.github.com/repos/opensyllabus/osp-scraper/pulls/124/comments","review_comment_url":"https://api.github.com/repos/opensyllabus/osp-scraper/pulls/comments{/number}","comments_url":"https://api.github.com/repos/opensyllabus/osp-scraper/issues/124/comments","statuses_url":"https://api.github.com/repos/opensyllabus/osp-scraper/statuses/79ffc0d0a5595d5f97f19fc1215172e117e66692","head":{"label":"opensyllabus:feat/make_filters","ref":"feat/make_filters","sha":"79ffc0d0a5595d5f97f19fc1215172e117e66692","user":{"login":"opensyllabus","id":7874404,"avatar_url":"https://avatars3.githubusercontent.com/u/7874404?v=3","gravatar_id":"","url":"https://api.github.com/users/opensyllabus","html_url":"https://github.com/opensyllabus","followers_url":"https://api.github.com/users/opensyllabus/followers","following_url":"https://api.github.com/users/opensyllabus/following{/other_user}","gists_url":"https://api.github.com/users/opensyllabus/gists{/gist_id}","starred_url":"https://api.github.com/users/opensyllabus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/opensyllabus/subscriptions","organizations_url":"https://api.github.com/users/opensyllabus/orgs","repos_url":"https://api.github.com/users/opensyllabus/repos","events_url":"https://api.github.com/users/opensyllabus/events{/privacy}","received_events_url":"https://api.github.com/users/opensyllabus/received_events","type":"Organization","site_admin":false},"repo":{"id":61742305,"name":"osp-scraper","full_name":"opensyllabus/osp-scraper","owner":{"login":"opensyllabus","id":7874404,"avatar_url":"https://avatars3.githubusercontent.com/u/7874404?v=3","gravatar_id":"","url":"https://api.github.com/users/opensyllabus","html_url":"https://github.com/opensyllabus","followers_url":"https://api.github.com/users/opensyllabus/followers","following_url":"https://api.github.com/users/opensyllabus/following{/other_user}","gists_url":"https://api.github.com/users/opensyllabus/gists{/gist_id}","starred_url":"https://api.github.com/users/opensyllabus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/opensyllabus/subscriptions","organizations_url":"https://api.github.com/users/opensyllabus/orgs","repos_url":"https://api.github.com/users/opensyllabus/repos","events_url":"https://api.github.com/users/opensyllabus/events{/privacy}","received_events_url":"https://api.github.com/users/opensyllabus/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/opensyllabus/osp-scraper","description":null,"fork":false,"url":"https://api.github.com/repos/opensyllabus/osp-scraper","forks_url":"https://api.github.com/repos/opensyllabus/osp-scraper/forks","keys_url":"https://api.github.com/repos/opensyllabus/osp-scraper/keys{/key_id}","collaborators_url":"https://api.github.com/repos/opensyllabus/osp-scraper/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/opensyllabus/osp-scraper/teams","hooks_url":"https://api.github.com/repos/opensyllabus/osp-scraper/hooks","issue_events_url":"https://api.github.com/repos/opensyllabus/osp-scraper/issues/events{/number}","events_url":"https://api.github.com/repos/opensyllabus/osp-scraper/events","assignees_url":"https://api.github.com/repos/opensyllabus/osp-scraper/assignees{/user}","branches_url":"https://api.github.com/repos/opensyllabus/osp-scraper/branches{/branch}","tags_url":"https://api.github.com/repos/opensyllabus/osp-scraper/tags","blobs_url":"https://api.github.com/repos/opensyllabus/osp-scraper/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/opensyllabus/osp-scraper/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/opensyllabus/osp-scraper/git/refs{/sha}","trees_url":"https://api.github.com/repos/opensyllabus/osp-scraper/git/trees{/sha}","statuses_url":"https://api.github.com/repos/opensyllabus/osp-scraper/statuses/{sha}","languages_url":"https://api.github.com/repos/opensyllabus/osp-scraper/languages","stargazers_url":"https://api.github.com/repos/opensyllabus/osp-scraper/stargazers","contributors_url":"https://api.github.com/repos/opensyllabus/osp-scraper/contributors","subscribers_url":"https://api.github.com/repos/opensyllabus/osp-scraper/subscribers","subscription_url":"https://api.github.com/repos/opensyllabus/osp-scraper/subscription","commits_url":"https://api.github.com/repos/opensyllabus/osp-scraper/commits{/sha}","git_commits_url":"https://api.github.com/repos/opensyllabus/osp-scraper/git/commits{/sha}","comments_url":"https://api.github.com/repos/opensyllabus/osp-scraper/comments{/number}","issue_comment_url":"https://api.github.com/repos/opensyllabus/osp-scraper/issues/comments{/number}","contents_url":"https://api.github.com/repos/opensyllabus/osp-scraper/contents/{+path}","compare_url":"https://api.github.com/repos/opensyllabus/osp-scraper/compare/{base}...{head}","merges_url":"https://api.github.com/repos/opensyllabus/osp-scraper/merges","archive_url":"https://api.github.com/repos/opensyllabus/osp-scraper/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/opensyllabus/osp-scraper/downloads","issues_url":"https://api.github.com/repos/opensyllabus/osp-scraper/issues{/number}","pulls_url":"https://api.github.com/repos/opensyllabus/osp-scraper/pulls{/number}","milestones_url":"https://api.github.com/repos/opensyllabus/osp-scraper/milestones{/number}","notifications_url":"https://api.github.com/repos/opensyllabus/osp-scraper/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/opensyllabus/osp-scraper/labels{/name}","releases_url":"https://api.github.com/repos/opensyllabus/osp-scraper/releases{/id}","deployments_url":"https://api.github.com/repos/opensyllabus/osp-scraper/deployments","created_at":"2016-06-22T18:30:40Z","updated_at":"2017-06-06T16:55:32Z","pushed_at":"2017-06-20T01:25:26Z","git_url":"git://github.com/opensyllabus/osp-scraper.git","ssh_url":"git@github.com:opensyllabus/osp-scraper.git","clone_url":"https://github.com/opensyllabus/osp-scraper.git","svn_url":"https://github.com/opensyllabus/osp-scraper","homepage":null,"size":360,"stargazers_count":3,"watchers_count":3,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":14,"forks":0,"open_issues":14,"watchers":3,"default_branch":"master"}},"base":{"label":"opensyllabus:master","ref":"master","sha":"4d390102547847a1f3ba8dae7636f426437688be","user":{"login":"opensyllabus","id":7874404,"avatar_url":"https://avatars3.githubusercontent.com/u/7874404?v=3","gravatar_id":"","url":"https://api.github.com/users/opensyllabus","html_url":"https://github.com/opensyllabus","followers_url":"https://api.github.com/users/opensyllabus/followers","following_url":"https://api.github.com/users/opensyllabus/following{/other_user}","gists_url":"https://api.github.com/users/opensyllabus/gists{/gist_id}","starred_url":"https://api.github.com/users/opensyllabus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/opensyllabus/subscriptions","organizations_url":"https://api.github.com/users/opensyllabus/orgs","repos_url":"https://api.github.com/users/opensyllabus/repos","events_url":"https://api.github.com/users/opensyllabus/events{/privacy}","received_events_url":"https://api.github.com/users/opensyllabus/received_events","type":"Organization","site_admin":false},"repo":{"id":61742305,"name":"osp-scraper","full_name":"opensyllabus/osp-scraper","owner":{"login":"opensyllabus","id":7874404,"avatar_url":"https://avatars3.githubusercontent.com/u/7874404?v=3","gravatar_id":"","url":"https://api.github.com/users/opensyllabus","html_url":"https://github.com/opensyllabus","followers_url":"https://api.github.com/users/opensyllabus/followers","following_url":"https://api.github.com/users/opensyllabus/following{/other_user}","gists_url":"https://api.github.com/users/opensyllabus/gists{/gist_id}","starred_url":"https://api.github.com/users/opensyllabus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/opensyllabus/subscriptions","organizations_url":"https://api.github.com/users/opensyllabus/orgs","repos_url":"https://api.github.com/users/opensyllabus/repos","events_url":"https://api.github.com/users/opensyllabus/events{/privacy}","received_events_url":"https://api.github.com/users/opensyllabus/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/opensyllabus/osp-scraper","description":null,"fork":false,"url":"https://api.github.com/repos/opensyllabus/osp-scraper","forks_url":"https://api.github.com/repos/opensyllabus/osp-scraper/forks","keys_url":"https://api.github.com/repos/opensyllabus/osp-scraper/keys{/key_id}","collaborators_url":"https://api.github.com/repos/opensyllabus/osp-scraper/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/opensyllabus/osp-scraper/teams","hooks_url":"https://api.github.com/repos/opensyllabus/osp-scraper/hooks","issue_events_url":"https://api.github.com/repos/opensyllabus/osp-scraper/issues/events{/number}","events_url":"https://api.github.com/repos/opensyllabus/osp-scraper/events","assignees_url":"https://api.github.com/repos/opensyllabus/osp-scraper/assignees{/user}","branches_url":"https://api.github.com/repos/opensyllabus/osp-scraper/branches{/branch}","tags_url":"https://api.github.com/repos/opensyllabus/osp-scraper/tags","blobs_url":"https://api.github.com/repos/opensyllabus/osp-scraper/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/opensyllabus/osp-scraper/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/opensyllabus/osp-scraper/git/refs{/sha}","trees_url":"https://api.github.com/repos/opensyllabus/osp-scraper/git/trees{/sha}","statuses_url":"https://api.github.com/repos/opensyllabus/osp-scraper/statuses/{sha}","languages_url":"https://api.github.com/repos/opensyllabus/osp-scraper/languages","stargazers_url":"https://api.github.com/repos/opensyllabus/osp-scraper/stargazers","contributors_url":"https://api.github.com/repos/opensyllabus/osp-scraper/contributors","subscribers_url":"https://api.github.com/repos/opensyllabus/osp-scraper/subscribers","subscription_url":"https://api.github.com/repos/opensyllabus/osp-scraper/subscription","commits_url":"https://api.github.com/repos/opensyllabus/osp-scraper/commits{/sha}","git_commits_url":"https://api.github.com/repos/opensyllabus/osp-scraper/git/commits{/sha}","comments_url":"https://api.github.com/repos/opensyllabus/osp-scraper/comments{/number}","issue_comment_url":"https://api.github.com/repos/opensyllabus/osp-scraper/issues/comments{/number}","contents_url":"https://api.github.com/repos/opensyllabus/osp-scraper/contents/{+path}","compare_url":"https://api.github.com/repos/opensyllabus/osp-scraper/compare/{base}...{head}","merges_url":"https://api.github.com/repos/opensyllabus/osp-scraper/merges","archive_url":"https://api.github.com/repos/opensyllabus/osp-scraper/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/opensyllabus/osp-scraper/downloads","issues_url":"https://api.github.com/repos/opensyllabus/osp-scraper/issues{/number}","pulls_url":"https://api.github.com/repos/opensyllabus/osp-scraper/pulls{/number}","milestones_url":"https://api.github.com/repos/opensyllabus/osp-scraper/milestones{/number}","notifications_url":"https://api.github.com/repos/opensyllabus/osp-scraper/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/opensyllabus/osp-scraper/labels{/name}","releases_url":"https://api.github.com/repos/opensyllabus/osp-scraper/releases{/id}","deployments_url":"https://api.github.com/repos/opensyllabus/osp-scraper/deployments","created_at":"2016-06-22T18:30:40Z","updated_at":"2017-06-06T16:55:32Z","pushed_at":"2017-06-20T01:25:26Z","git_url":"git://github.com/opensyllabus/osp-scraper.git","ssh_url":"git@github.com:opensyllabus/osp-scraper.git","clone_url":"https://github.com/opensyllabus/osp-scraper.git","svn_url":"https://github.com/opensyllabus/osp-scraper","homepage":null,"size":360,"stargazers_count":3,"watchers_count":3,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":14,"forks":0,"open_issues":14,"watchers":3,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/opensyllabus/osp-scraper/pulls/124"},"html":{"href":"https://github.com/opensyllabus/osp-scraper/pull/124"},"issue":{"href":"https://api.github.com/repos/opensyllabus/osp-scraper/issues/124"},"comments":{"href":"https://api.github.com/repos/opensyllabus/osp-scraper/issues/124/comments"},"review_comments":{"href":"https://api.github.com/repos/opensyllabus/osp-scraper/pulls/124/comments"},"review_comment":{"href":"https://api.github.com/repos/opensyllabus/osp-scraper/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/opensyllabus/osp-scraper/pulls/124/commits"},"statuses":{"href":"https://api.github.com/repos/opensyllabus/osp-scraper/statuses/79ffc0d0a5595d5f97f19fc1215172e117e66692"}}}}
{"actor":{"display_login":"HarrisonGregg"}}