{"action":"created","comment":{"url":"https://api.github.com/repos/18F/domain-scan/pulls/comments/117645560","pull_request_review_id":39368015,"id":117645560,"diff_hunk":"@@ -49,7 +49,7 @@\n wildcard_pattern = re.compile(\"^\\*\\.\")\n redacted_pattern = re.compile(\"^(\\?\\.)+\")\n \n-def gather(suffix, options):\n+def gather(suffix, options, extra={}):","path":"gatherers/censys.py","position":5,"original_position":5,"commit_id":"29efc680a4bbb184bb88b5ebde0c5b62ba2ad5ca","original_commit_id":"29efc680a4bbb184bb88b5ebde0c5b62ba2ad5ca","user":{"login":"konklone","id":4592,"avatar_url":"https://avatars3.githubusercontent.com/u/4592?v=3","gravatar_id":"","url":"https://api.github.com/users/konklone","html_url":"https://github.com/konklone","followers_url":"https://api.github.com/users/konklone/followers","following_url":"https://api.github.com/users/konklone/following{/other_user}","gists_url":"https://api.github.com/users/konklone/gists{/gist_id}","starred_url":"https://api.github.com/users/konklone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/konklone/subscriptions","organizations_url":"https://api.github.com/users/konklone/orgs","repos_url":"https://api.github.com/users/konklone/repos","events_url":"https://api.github.com/users/konklone/events{/privacy}","received_events_url":"https://api.github.com/users/konklone/received_events","type":"User","site_admin":false},"body":"The ability to use the `url` gatherer under different names meant changing the method signature for each gatherer's `gather` function to include an optional `extra` dict, which might have information that doesn't appear directly in the command line flags. \r\n\r\nIn this case, the `url` gatherer's `gather` method has to know, at call-time, which of the names it's operating under, so that it can pull the right CSV URL/path from the right command line flag.","created_at":"2017-05-21T23:06:29Z","updated_at":"2017-05-21T23:08:08Z","html_url":"https://github.com/18F/domain-scan/pull/124#discussion_r117645560","pull_request_url":"https://api.github.com/repos/18F/domain-scan/pulls/124","_links":{"self":{"href":"https://api.github.com/repos/18F/domain-scan/pulls/comments/117645560"},"html":{"href":"https://github.com/18F/domain-scan/pull/124#discussion_r117645560"},"pull_request":{"href":"https://api.github.com/repos/18F/domain-scan/pulls/124"}}},"pull_request":{"url":"https://api.github.com/repos/18F/domain-scan/pulls/124","id":121679643,"html_url":"https://github.com/18F/domain-scan/pull/124","diff_url":"https://github.com/18F/domain-scan/pull/124.diff","patch_url":"https://github.com/18F/domain-scan/pull/124.patch","issue_url":"https://api.github.com/repos/18F/domain-scan/issues/124","number":124,"state":"open","locked":false,"title":"Gathering from multiple sources at once","user":{"login":"konklone","id":4592,"avatar_url":"https://avatars3.githubusercontent.com/u/4592?v=3","gravatar_id":"","url":"https://api.github.com/users/konklone","html_url":"https://github.com/konklone","followers_url":"https://api.github.com/users/konklone/followers","following_url":"https://api.github.com/users/konklone/following{/other_user}","gists_url":"https://api.github.com/users/konklone/gists{/gist_id}","starred_url":"https://api.github.com/users/konklone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/konklone/subscriptions","organizations_url":"https://api.github.com/users/konklone/orgs","repos_url":"https://api.github.com/users/konklone/repos","events_url":"https://api.github.com/users/konklone/events{/privacy}","received_events_url":"https://api.github.com/users/konklone/received_events","type":"User","site_admin":false},"body":"This pull request adds a new way of gathering hostnames from multiple sources at once. Results are collectively deduped, and the resulting CSV will show whether each hostname was found in each checked source.\r\n\r\nAn example (real-world) use of this facility, to gather publicly discoverable federal .gov hostnames from Censys.io, the Digital Analytics Program, the 2016 End of Term Archive, and the official .gov domain list:\r\n\r\n```bash\r\n./gather censys,dap,eot2016,parents --eot2016=https://github.com/GSA/data/raw/gh-pages/end-of-term-archive-csv/eot-2016-seeds.csv --dap=https://analytics.usa.gov/data/live/sites-extended.csv --suffix=.gov --parents=https://raw.githubusercontent.com/GSA/data/gh-pages/dotgov-domains/current-federal.csv --include-parents --debug --export\r\n```\r\nAs shown in this example, simple URL/file-based gatherers are dynamically created by giving them a name that isn't already defined as a gatherer (right now, anything other than `censys`), and then specified by using a command line flag with the same name. This dynamic approach is used to keep syntax simple and avoid having to use more complicated syntax (such as numerically sequenced names), while allowing multiple URL/file-based gatherers to run separately without their namespaces colliding.","created_at":"2017-05-21T23:01:27Z","updated_at":"2017-05-21T23:08:08Z","closed_at":null,"merged_at":null,"merge_commit_sha":"9c195c772e62453597df70f7c76fb23add656b69","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/18F/domain-scan/pulls/124/commits","review_comments_url":"https://api.github.com/repos/18F/domain-scan/pulls/124/comments","review_comment_url":"https://api.github.com/repos/18F/domain-scan/pulls/comments{/number}","comments_url":"https://api.github.com/repos/18F/domain-scan/issues/124/comments","statuses_url":"https://api.github.com/repos/18F/domain-scan/statuses/29efc680a4bbb184bb88b5ebde0c5b62ba2ad5ca","head":{"label":"18F:multiple-gatherers","ref":"multiple-gatherers","sha":"29efc680a4bbb184bb88b5ebde0c5b62ba2ad5ca","user":{"login":"18F","id":6233994,"avatar_url":"https://avatars3.githubusercontent.com/u/6233994?v=3","gravatar_id":"","url":"https://api.github.com/users/18F","html_url":"https://github.com/18F","followers_url":"https://api.github.com/users/18F/followers","following_url":"https://api.github.com/users/18F/following{/other_user}","gists_url":"https://api.github.com/users/18F/gists{/gist_id}","starred_url":"https://api.github.com/users/18F/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/18F/subscriptions","organizations_url":"https://api.github.com/users/18F/orgs","repos_url":"https://api.github.com/users/18F/repos","events_url":"https://api.github.com/users/18F/events{/privacy}","received_events_url":"https://api.github.com/users/18F/received_events","type":"Organization","site_admin":false},"repo":{"id":27473797,"name":"domain-scan","full_name":"18F/domain-scan","owner":{"login":"18F","id":6233994,"avatar_url":"https://avatars3.githubusercontent.com/u/6233994?v=3","gravatar_id":"","url":"https://api.github.com/users/18F","html_url":"https://github.com/18F","followers_url":"https://api.github.com/users/18F/followers","following_url":"https://api.github.com/users/18F/following{/other_user}","gists_url":"https://api.github.com/users/18F/gists{/gist_id}","starred_url":"https://api.github.com/users/18F/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/18F/subscriptions","organizations_url":"https://api.github.com/users/18F/orgs","repos_url":"https://api.github.com/users/18F/repos","events_url":"https://api.github.com/users/18F/events{/privacy}","received_events_url":"https://api.github.com/users/18F/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/18F/domain-scan","description":"A standard pipeline for running open source scanning tools on domains to measure things like speed, accessibiity, and HTTPS.","fork":false,"url":"https://api.github.com/repos/18F/domain-scan","forks_url":"https://api.github.com/repos/18F/domain-scan/forks","keys_url":"https://api.github.com/repos/18F/domain-scan/keys{/key_id}","collaborators_url":"https://api.github.com/repos/18F/domain-scan/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/18F/domain-scan/teams","hooks_url":"https://api.github.com/repos/18F/domain-scan/hooks","issue_events_url":"https://api.github.com/repos/18F/domain-scan/issues/events{/number}","events_url":"https://api.github.com/repos/18F/domain-scan/events","assignees_url":"https://api.github.com/repos/18F/domain-scan/assignees{/user}","branches_url":"https://api.github.com/repos/18F/domain-scan/branches{/branch}","tags_url":"https://api.github.com/repos/18F/domain-scan/tags","blobs_url":"https://api.github.com/repos/18F/domain-scan/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/18F/domain-scan/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/18F/domain-scan/git/refs{/sha}","trees_url":"https://api.github.com/repos/18F/domain-scan/git/trees{/sha}","statuses_url":"https://api.github.com/repos/18F/domain-scan/statuses/{sha}","languages_url":"https://api.github.com/repos/18F/domain-scan/languages","stargazers_url":"https://api.github.com/repos/18F/domain-scan/stargazers","contributors_url":"https://api.github.com/repos/18F/domain-scan/contributors","subscribers_url":"https://api.github.com/repos/18F/domain-scan/subscribers","subscription_url":"https://api.github.com/repos/18F/domain-scan/subscription","commits_url":"https://api.github.com/repos/18F/domain-scan/commits{/sha}","git_commits_url":"https://api.github.com/repos/18F/domain-scan/git/commits{/sha}","comments_url":"https://api.github.com/repos/18F/domain-scan/comments{/number}","issue_comment_url":"https://api.github.com/repos/18F/domain-scan/issues/comments{/number}","contents_url":"https://api.github.com/repos/18F/domain-scan/contents/{+path}","compare_url":"https://api.github.com/repos/18F/domain-scan/compare/{base}...{head}","merges_url":"https://api.github.com/repos/18F/domain-scan/merges","archive_url":"https://api.github.com/repos/18F/domain-scan/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/18F/domain-scan/downloads","issues_url":"https://api.github.com/repos/18F/domain-scan/issues{/number}","pulls_url":"https://api.github.com/repos/18F/domain-scan/pulls{/number}","milestones_url":"https://api.github.com/repos/18F/domain-scan/milestones{/number}","notifications_url":"https://api.github.com/repos/18F/domain-scan/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/18F/domain-scan/labels{/name}","releases_url":"https://api.github.com/repos/18F/domain-scan/releases{/id}","deployments_url":"https://api.github.com/repos/18F/domain-scan/deployments","created_at":"2014-12-03T06:52:17Z","updated_at":"2017-05-16T19:44:21Z","pushed_at":"2017-05-21T23:01:28Z","git_url":"git://github.com/18F/domain-scan.git","ssh_url":"git@github.com:18F/domain-scan.git","clone_url":"https://github.com/18F/domain-scan.git","svn_url":"https://github.com/18F/domain-scan","homepage":"","size":6868,"stargazers_count":113,"watchers_count":113,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":46,"mirror_url":null,"open_issues_count":22,"forks":46,"open_issues":22,"watchers":113,"default_branch":"master"}},"base":{"label":"18F:master","ref":"master","sha":"196845e22d6bea9224d42cc2d2f86b0de3429c6f","user":{"login":"18F","id":6233994,"avatar_url":"https://avatars3.githubusercontent.com/u/6233994?v=3","gravatar_id":"","url":"https://api.github.com/users/18F","html_url":"https://github.com/18F","followers_url":"https://api.github.com/users/18F/followers","following_url":"https://api.github.com/users/18F/following{/other_user}","gists_url":"https://api.github.com/users/18F/gists{/gist_id}","starred_url":"https://api.github.com/users/18F/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/18F/subscriptions","organizations_url":"https://api.github.com/users/18F/orgs","repos_url":"https://api.github.com/users/18F/repos","events_url":"https://api.github.com/users/18F/events{/privacy}","received_events_url":"https://api.github.com/users/18F/received_events","type":"Organization","site_admin":false},"repo":{"id":27473797,"name":"domain-scan","full_name":"18F/domain-scan","owner":{"login":"18F","id":6233994,"avatar_url":"https://avatars3.githubusercontent.com/u/6233994?v=3","gravatar_id":"","url":"https://api.github.com/users/18F","html_url":"https://github.com/18F","followers_url":"https://api.github.com/users/18F/followers","following_url":"https://api.github.com/users/18F/following{/other_user}","gists_url":"https://api.github.com/users/18F/gists{/gist_id}","starred_url":"https://api.github.com/users/18F/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/18F/subscriptions","organizations_url":"https://api.github.com/users/18F/orgs","repos_url":"https://api.github.com/users/18F/repos","events_url":"https://api.github.com/users/18F/events{/privacy}","received_events_url":"https://api.github.com/users/18F/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/18F/domain-scan","description":"A standard pipeline for running open source scanning tools on domains to measure things like speed, accessibiity, and HTTPS.","fork":false,"url":"https://api.github.com/repos/18F/domain-scan","forks_url":"https://api.github.com/repos/18F/domain-scan/forks","keys_url":"https://api.github.com/repos/18F/domain-scan/keys{/key_id}","collaborators_url":"https://api.github.com/repos/18F/domain-scan/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/18F/domain-scan/teams","hooks_url":"https://api.github.com/repos/18F/domain-scan/hooks","issue_events_url":"https://api.github.com/repos/18F/domain-scan/issues/events{/number}","events_url":"https://api.github.com/repos/18F/domain-scan/events","assignees_url":"https://api.github.com/repos/18F/domain-scan/assignees{/user}","branches_url":"https://api.github.com/repos/18F/domain-scan/branches{/branch}","tags_url":"https://api.github.com/repos/18F/domain-scan/tags","blobs_url":"https://api.github.com/repos/18F/domain-scan/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/18F/domain-scan/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/18F/domain-scan/git/refs{/sha}","trees_url":"https://api.github.com/repos/18F/domain-scan/git/trees{/sha}","statuses_url":"https://api.github.com/repos/18F/domain-scan/statuses/{sha}","languages_url":"https://api.github.com/repos/18F/domain-scan/languages","stargazers_url":"https://api.github.com/repos/18F/domain-scan/stargazers","contributors_url":"https://api.github.com/repos/18F/domain-scan/contributors","subscribers_url":"https://api.github.com/repos/18F/domain-scan/subscribers","subscription_url":"https://api.github.com/repos/18F/domain-scan/subscription","commits_url":"https://api.github.com/repos/18F/domain-scan/commits{/sha}","git_commits_url":"https://api.github.com/repos/18F/domain-scan/git/commits{/sha}","comments_url":"https://api.github.com/repos/18F/domain-scan/comments{/number}","issue_comment_url":"https://api.github.com/repos/18F/domain-scan/issues/comments{/number}","contents_url":"https://api.github.com/repos/18F/domain-scan/contents/{+path}","compare_url":"https://api.github.com/repos/18F/domain-scan/compare/{base}...{head}","merges_url":"https://api.github.com/repos/18F/domain-scan/merges","archive_url":"https://api.github.com/repos/18F/domain-scan/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/18F/domain-scan/downloads","issues_url":"https://api.github.com/repos/18F/domain-scan/issues{/number}","pulls_url":"https://api.github.com/repos/18F/domain-scan/pulls{/number}","milestones_url":"https://api.github.com/repos/18F/domain-scan/milestones{/number}","notifications_url":"https://api.github.com/repos/18F/domain-scan/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/18F/domain-scan/labels{/name}","releases_url":"https://api.github.com/repos/18F/domain-scan/releases{/id}","deployments_url":"https://api.github.com/repos/18F/domain-scan/deployments","created_at":"2014-12-03T06:52:17Z","updated_at":"2017-05-16T19:44:21Z","pushed_at":"2017-05-21T23:01:28Z","git_url":"git://github.com/18F/domain-scan.git","ssh_url":"git@github.com:18F/domain-scan.git","clone_url":"https://github.com/18F/domain-scan.git","svn_url":"https://github.com/18F/domain-scan","homepage":"","size":6868,"stargazers_count":113,"watchers_count":113,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":46,"mirror_url":null,"open_issues_count":22,"forks":46,"open_issues":22,"watchers":113,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/18F/domain-scan/pulls/124"},"html":{"href":"https://github.com/18F/domain-scan/pull/124"},"issue":{"href":"https://api.github.com/repos/18F/domain-scan/issues/124"},"comments":{"href":"https://api.github.com/repos/18F/domain-scan/issues/124/comments"},"review_comments":{"href":"https://api.github.com/repos/18F/domain-scan/pulls/124/comments"},"review_comment":{"href":"https://api.github.com/repos/18F/domain-scan/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/18F/domain-scan/pulls/124/commits"},"statuses":{"href":"https://api.github.com/repos/18F/domain-scan/statuses/29efc680a4bbb184bb88b5ebde0c5b62ba2ad5ca"}}}}
{"actor":{"display_login":"konklone"}}