{"action":"created","comment":{"url":"https://api.github.com/repos/18F/domain-scan/pulls/comments/150393357","pull_request_review_id":75941590,"id":150393357,"diff_hunk":"@@ -1,82 +1,85 @@\n import logging\n-from scanners import utils\n-import os\n-import json\n-import glob\n \n-###\n-# == pshtt ==\n-#\n-# Inspect a site's TLS configuration using DHS NCATS' pshtt tool.\n-#\n-###\n+import re\n+from pshtt import pshtt\n \n-command = os.environ.get(\"PSHTT_PATH\", \"pshtt\")","path":"scanners/pshtt.py","position":16,"original_position":16,"commit_id":"f9456ee141b124f590c9600bbed4f8b919be688e","original_commit_id":"f9456ee141b124f590c9600bbed4f8b919be688e","user":{"login":"konklone","id":4592,"avatar_url":"https://avatars0.githubusercontent.com/u/4592?v=4","gravatar_id":"","url":"https://api.github.com/users/konklone","html_url":"https://github.com/konklone","followers_url":"https://api.github.com/users/konklone/followers","following_url":"https://api.github.com/users/konklone/following{/other_user}","gists_url":"https://api.github.com/users/konklone/gists{/gist_id}","starred_url":"https://api.github.com/users/konklone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/konklone/subscriptions","organizations_url":"https://api.github.com/users/konklone/orgs","repos_url":"https://api.github.com/users/konklone/repos","events_url":"https://api.github.com/users/konklone/events{/privacy}","received_events_url":"https://api.github.com/users/konklone/received_events","type":"User","site_admin":false},"body":"In addition to porting the pshtt scanner to work on Lambda, I also ported it to use the Python API rather than shelling out to `pshtt`. We don't reference a `command` or environment variables for anything.","created_at":"2017-11-11T20:56:14Z","updated_at":"2017-11-11T20:59:08Z","html_url":"https://github.com/18F/domain-scan/pull/155#discussion_r150393357","pull_request_url":"https://api.github.com/repos/18F/domain-scan/pulls/155","author_association":"OWNER","_links":{"self":{"href":"https://api.github.com/repos/18F/domain-scan/pulls/comments/150393357"},"html":{"href":"https://github.com/18F/domain-scan/pull/155#discussion_r150393357"},"pull_request":{"href":"https://api.github.com/repos/18F/domain-scan/pulls/155"}}},"pull_request":{"url":"https://api.github.com/repos/18F/domain-scan/pulls/155","id":150715564,"html_url":"https://github.com/18F/domain-scan/pull/155","diff_url":"https://github.com/18F/domain-scan/pull/155.diff","patch_url":"https://github.com/18F/domain-scan/pull/155.patch","issue_url":"https://api.github.com/repos/18F/domain-scan/issues/155","number":155,"state":"open","locked":false,"title":"Scanning pipeline refactor, and a Lambda-based pipeline","user":{"login":"konklone","id":4592,"avatar_url":"https://avatars0.githubusercontent.com/u/4592?v=4","gravatar_id":"","url":"https://api.github.com/users/konklone","html_url":"https://github.com/konklone","followers_url":"https://api.github.com/users/konklone/followers","following_url":"https://api.github.com/users/konklone/following{/other_user}","gists_url":"https://api.github.com/users/konklone/gists{/gist_id}","starred_url":"https://api.github.com/users/konklone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/konklone/subscriptions","organizations_url":"https://api.github.com/users/konklone/orgs","repos_url":"https://api.github.com/users/konklone/repos","events_url":"https://api.github.com/users/konklone/events{/privacy}","received_events_url":"https://api.github.com/users/konklone/received_events","type":"User","site_admin":false},"body":"This is a very large PR, as my attempt to introduce a Lambda-based pipeline also led me to a long-overdue refactor of the overall scanning pipeline. \r\n\r\nThe Lambda pipeline still relies on one local worker thread per Lambda worker, so there is some local bottlenecking happening. However, local worker threads that are waiting around for Lambda responses are doing much less work than worker threads that are running scans, and so it should be possible to run many more workers when using the Lambda pipeline. (I'm still evaluating the real effects when running large scans.) Local systems may need to expand their system's allowed max number of processes in order to take full advantage.\r\n\r\nFuture expansions to this pipeline could include a cloud-based queueing system, so that local execution is single-threaded and all parallel activity happens in the cloud. For now, the main benefits of the Lambda pipeline are realized in expanding the number of worker thread substantially, and in eliminating the effects of memory leaks during long-running scans (such as SSLyze's memory leaks when running in synchronous mode).\r\n\r\nSome of the general changes:\r\n\r\n* All caching of scan data is now performed centrally -- scanners don't need to manage their own caching logic.\r\n* While data is always stored in the `cache/` directory after a scan, the use of cached data to skip the execution of a scan is now **opt-in** instead of **opt-out**. The `--force` flag has been replaced with a `--cache` flag, which enables the use of cached data to skip scans. Use of cached data to skip scans is almost always used only in debugging and development, not in production, so this felt like it made sense to do.\r\n* Scanners can now define an `init` method that gets run for all scans of that type, and an `init_domain` method that gets run for each scan. Both methods can return dicts that will get merged and passed to the scanner's `scan` method as an `environment` dict. The `init_domain` method can return `False` to declare a domain ineligible for that scan and cause it to be skipped.\r\n* A scanner's `scan` method now returns a dict of data, rather than one or more rows. This dict is what is serialized to JSON and cached for that method, so every scanner is responsible for assembling a dict of data to be cached.\r\n* A scanner must now define a `to_rows` method that, given the dict it returned, will transform that dict into one or more rows for CSV output. This decouples CSV-specific logic from other scanning logic.\r\n* A `--meta` flag that causes the resulting CSVs to append extra fields with information about the scan, such as encountered errors and timing information.\r\n* Extra duration and scan info is stored in `meta.json`, including a breakdown per-scanner.\r\n\r\nThe Lambda pipeline:\r\n\r\n* To execute a scan inside of Lambda instead of locally, add the `--lambda` flag. It's (sort of) that simple!\r\n* When using the Lambda pipeline, a scanner's `scan` method will be run inside Lambda. However, all other methods (`init`, `init_domain`, `to_rows`) will be run locally.\r\n* Lambda functions need to be deployed to a specific AWS account before they can be run. Some build scripts to do so, along with a precompiled virtualenv with all required dependencies, are included. Documentation will be added before merge.\r\n* If the Lambda pipeline is being used, `--meta` will also append a bunch of Lambda-related fields, such as the request ID, logging information, and timing information.\r\n* If `--meta` and `--lambda` are passed, `--lambda-details` will cause the scanner to fetch even more post-scan data from Lambda CloudWatch logs, after waiting 20 seconds to allow logs to appear in CloudWatch. This will make one log request per-scan, so can be quite expensive. However, it's the only way to track how much memory individual scan executions actually used.\r\n\r\nTODO:\r\n\r\n- [ ] Add documentation, including for the Lambda deploy pipeline.\r\n- [ ] Update `lambda/deploy` to have a `--create` mode that uses the create-function method the first time.\r\n- [ ] Update all scanners to use the new format.\r\n- [ ] Remove unused scanners (e.g. `tls`).\r\n- [ ] Slight refactor of local/lambda branching execution code to reduce duplicated code between both scan methods.\r\n- [ ] Make sure Dockerfile still works.","created_at":"2017-11-05T10:03:09Z","updated_at":"2017-11-11T20:59:08Z","closed_at":null,"merged_at":null,"merge_commit_sha":"848e981c044e234d3b60945b733f4a06079a9ba2","assignee":null,"assignees":[],"requested_reviewers":[],"milestone":null,"commits_url":"https://api.github.com/repos/18F/domain-scan/pulls/155/commits","review_comments_url":"https://api.github.com/repos/18F/domain-scan/pulls/155/comments","review_comment_url":"https://api.github.com/repos/18F/domain-scan/pulls/comments{/number}","comments_url":"https://api.github.com/repos/18F/domain-scan/issues/155/comments","statuses_url":"https://api.github.com/repos/18F/domain-scan/statuses/f9456ee141b124f590c9600bbed4f8b919be688e","head":{"label":"18F:exploring-lambda","ref":"exploring-lambda","sha":"f9456ee141b124f590c9600bbed4f8b919be688e","user":{"login":"18F","id":6233994,"avatar_url":"https://avatars0.githubusercontent.com/u/6233994?v=4","gravatar_id":"","url":"https://api.github.com/users/18F","html_url":"https://github.com/18F","followers_url":"https://api.github.com/users/18F/followers","following_url":"https://api.github.com/users/18F/following{/other_user}","gists_url":"https://api.github.com/users/18F/gists{/gist_id}","starred_url":"https://api.github.com/users/18F/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/18F/subscriptions","organizations_url":"https://api.github.com/users/18F/orgs","repos_url":"https://api.github.com/users/18F/repos","events_url":"https://api.github.com/users/18F/events{/privacy}","received_events_url":"https://api.github.com/users/18F/received_events","type":"Organization","site_admin":false},"repo":{"id":27473797,"name":"domain-scan","full_name":"18F/domain-scan","owner":{"login":"18F","id":6233994,"avatar_url":"https://avatars0.githubusercontent.com/u/6233994?v=4","gravatar_id":"","url":"https://api.github.com/users/18F","html_url":"https://github.com/18F","followers_url":"https://api.github.com/users/18F/followers","following_url":"https://api.github.com/users/18F/following{/other_user}","gists_url":"https://api.github.com/users/18F/gists{/gist_id}","starred_url":"https://api.github.com/users/18F/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/18F/subscriptions","organizations_url":"https://api.github.com/users/18F/orgs","repos_url":"https://api.github.com/users/18F/repos","events_url":"https://api.github.com/users/18F/events{/privacy}","received_events_url":"https://api.github.com/users/18F/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/18F/domain-scan","description":"A pipeline for scanning domains to measure things like speed, accessibility, and HTTPS.","fork":false,"url":"https://api.github.com/repos/18F/domain-scan","forks_url":"https://api.github.com/repos/18F/domain-scan/forks","keys_url":"https://api.github.com/repos/18F/domain-scan/keys{/key_id}","collaborators_url":"https://api.github.com/repos/18F/domain-scan/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/18F/domain-scan/teams","hooks_url":"https://api.github.com/repos/18F/domain-scan/hooks","issue_events_url":"https://api.github.com/repos/18F/domain-scan/issues/events{/number}","events_url":"https://api.github.com/repos/18F/domain-scan/events","assignees_url":"https://api.github.com/repos/18F/domain-scan/assignees{/user}","branches_url":"https://api.github.com/repos/18F/domain-scan/branches{/branch}","tags_url":"https://api.github.com/repos/18F/domain-scan/tags","blobs_url":"https://api.github.com/repos/18F/domain-scan/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/18F/domain-scan/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/18F/domain-scan/git/refs{/sha}","trees_url":"https://api.github.com/repos/18F/domain-scan/git/trees{/sha}","statuses_url":"https://api.github.com/repos/18F/domain-scan/statuses/{sha}","languages_url":"https://api.github.com/repos/18F/domain-scan/languages","stargazers_url":"https://api.github.com/repos/18F/domain-scan/stargazers","contributors_url":"https://api.github.com/repos/18F/domain-scan/contributors","subscribers_url":"https://api.github.com/repos/18F/domain-scan/subscribers","subscription_url":"https://api.github.com/repos/18F/domain-scan/subscription","commits_url":"https://api.github.com/repos/18F/domain-scan/commits{/sha}","git_commits_url":"https://api.github.com/repos/18F/domain-scan/git/commits{/sha}","comments_url":"https://api.github.com/repos/18F/domain-scan/comments{/number}","issue_comment_url":"https://api.github.com/repos/18F/domain-scan/issues/comments{/number}","contents_url":"https://api.github.com/repos/18F/domain-scan/contents/{+path}","compare_url":"https://api.github.com/repos/18F/domain-scan/compare/{base}...{head}","merges_url":"https://api.github.com/repos/18F/domain-scan/merges","archive_url":"https://api.github.com/repos/18F/domain-scan/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/18F/domain-scan/downloads","issues_url":"https://api.github.com/repos/18F/domain-scan/issues{/number}","pulls_url":"https://api.github.com/repos/18F/domain-scan/pulls{/number}","milestones_url":"https://api.github.com/repos/18F/domain-scan/milestones{/number}","notifications_url":"https://api.github.com/repos/18F/domain-scan/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/18F/domain-scan/labels{/name}","releases_url":"https://api.github.com/repos/18F/domain-scan/releases{/id}","deployments_url":"https://api.github.com/repos/18F/domain-scan/deployments","created_at":"2014-12-03T06:52:17Z","updated_at":"2017-11-06T22:05:31Z","pushed_at":"2017-11-11T20:46:17Z","git_url":"git://github.com/18F/domain-scan.git","ssh_url":"git@github.com:18F/domain-scan.git","clone_url":"https://github.com/18F/domain-scan.git","svn_url":"https://github.com/18F/domain-scan","homepage":"","size":50140,"stargazers_count":145,"watchers_count":145,"language":"Python","has_issues":true,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":62,"mirror_url":null,"archived":false,"open_issues_count":13,"forks":62,"open_issues":13,"watchers":145,"default_branch":"master"}},"base":{"label":"18F:master","ref":"master","sha":"d7c954fdaf171845148a3da12220add0dc8a9bf1","user":{"login":"18F","id":6233994,"avatar_url":"https://avatars0.githubusercontent.com/u/6233994?v=4","gravatar_id":"","url":"https://api.github.com/users/18F","html_url":"https://github.com/18F","followers_url":"https://api.github.com/users/18F/followers","following_url":"https://api.github.com/users/18F/following{/other_user}","gists_url":"https://api.github.com/users/18F/gists{/gist_id}","starred_url":"https://api.github.com/users/18F/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/18F/subscriptions","organizations_url":"https://api.github.com/users/18F/orgs","repos_url":"https://api.github.com/users/18F/repos","events_url":"https://api.github.com/users/18F/events{/privacy}","received_events_url":"https://api.github.com/users/18F/received_events","type":"Organization","site_admin":false},"repo":{"id":27473797,"name":"domain-scan","full_name":"18F/domain-scan","owner":{"login":"18F","id":6233994,"avatar_url":"https://avatars0.githubusercontent.com/u/6233994?v=4","gravatar_id":"","url":"https://api.github.com/users/18F","html_url":"https://github.com/18F","followers_url":"https://api.github.com/users/18F/followers","following_url":"https://api.github.com/users/18F/following{/other_user}","gists_url":"https://api.github.com/users/18F/gists{/gist_id}","starred_url":"https://api.github.com/users/18F/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/18F/subscriptions","organizations_url":"https://api.github.com/users/18F/orgs","repos_url":"https://api.github.com/users/18F/repos","events_url":"https://api.github.com/users/18F/events{/privacy}","received_events_url":"https://api.github.com/users/18F/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/18F/domain-scan","description":"A pipeline for scanning domains to measure things like speed, accessibility, and HTTPS.","fork":false,"url":"https://api.github.com/repos/18F/domain-scan","forks_url":"https://api.github.com/repos/18F/domain-scan/forks","keys_url":"https://api.github.com/repos/18F/domain-scan/keys{/key_id}","collaborators_url":"https://api.github.com/repos/18F/domain-scan/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/18F/domain-scan/teams","hooks_url":"https://api.github.com/repos/18F/domain-scan/hooks","issue_events_url":"https://api.github.com/repos/18F/domain-scan/issues/events{/number}","events_url":"https://api.github.com/repos/18F/domain-scan/events","assignees_url":"https://api.github.com/repos/18F/domain-scan/assignees{/user}","branches_url":"https://api.github.com/repos/18F/domain-scan/branches{/branch}","tags_url":"https://api.github.com/repos/18F/domain-scan/tags","blobs_url":"https://api.github.com/repos/18F/domain-scan/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/18F/domain-scan/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/18F/domain-scan/git/refs{/sha}","trees_url":"https://api.github.com/repos/18F/domain-scan/git/trees{/sha}","statuses_url":"https://api.github.com/repos/18F/domain-scan/statuses/{sha}","languages_url":"https://api.github.com/repos/18F/domain-scan/languages","stargazers_url":"https://api.github.com/repos/18F/domain-scan/stargazers","contributors_url":"https://api.github.com/repos/18F/domain-scan/contributors","subscribers_url":"https://api.github.com/repos/18F/domain-scan/subscribers","subscription_url":"https://api.github.com/repos/18F/domain-scan/subscription","commits_url":"https://api.github.com/repos/18F/domain-scan/commits{/sha}","git_commits_url":"https://api.github.com/repos/18F/domain-scan/git/commits{/sha}","comments_url":"https://api.github.com/repos/18F/domain-scan/comments{/number}","issue_comment_url":"https://api.github.com/repos/18F/domain-scan/issues/comments{/number}","contents_url":"https://api.github.com/repos/18F/domain-scan/contents/{+path}","compare_url":"https://api.github.com/repos/18F/domain-scan/compare/{base}...{head}","merges_url":"https://api.github.com/repos/18F/domain-scan/merges","archive_url":"https://api.github.com/repos/18F/domain-scan/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/18F/domain-scan/downloads","issues_url":"https://api.github.com/repos/18F/domain-scan/issues{/number}","pulls_url":"https://api.github.com/repos/18F/domain-scan/pulls{/number}","milestones_url":"https://api.github.com/repos/18F/domain-scan/milestones{/number}","notifications_url":"https://api.github.com/repos/18F/domain-scan/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/18F/domain-scan/labels{/name}","releases_url":"https://api.github.com/repos/18F/domain-scan/releases{/id}","deployments_url":"https://api.github.com/repos/18F/domain-scan/deployments","created_at":"2014-12-03T06:52:17Z","updated_at":"2017-11-06T22:05:31Z","pushed_at":"2017-11-11T20:46:17Z","git_url":"git://github.com/18F/domain-scan.git","ssh_url":"git@github.com:18F/domain-scan.git","clone_url":"https://github.com/18F/domain-scan.git","svn_url":"https://github.com/18F/domain-scan","homepage":"","size":50140,"stargazers_count":145,"watchers_count":145,"language":"Python","has_issues":true,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":62,"mirror_url":null,"archived":false,"open_issues_count":13,"forks":62,"open_issues":13,"watchers":145,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/18F/domain-scan/pulls/155"},"html":{"href":"https://github.com/18F/domain-scan/pull/155"},"issue":{"href":"https://api.github.com/repos/18F/domain-scan/issues/155"},"comments":{"href":"https://api.github.com/repos/18F/domain-scan/issues/155/comments"},"review_comments":{"href":"https://api.github.com/repos/18F/domain-scan/pulls/155/comments"},"review_comment":{"href":"https://api.github.com/repos/18F/domain-scan/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/18F/domain-scan/pulls/155/commits"},"statuses":{"href":"https://api.github.com/repos/18F/domain-scan/statuses/f9456ee141b124f590c9600bbed4f8b919be688e"}},"author_association":"OWNER"}}
{"actor":{"display_login":"konklone"}}