{"action":"created","comment":{"url":"https://api.github.com/repos/pantsbuild/pants/pulls/comments/99345256","pull_request_review_id":20020790,"id":99345256,"diff_hunk":"@@ -515,43 +562,33 @@ pub struct Task {\n }\n \n impl Step for Task {\n-  fn step(&self, context: StepContext) -> State<Node> {\n-    // Compute dependencies for the Node, or determine whether it is a Noop.\n-    let mut dependencies = Vec::new();\n-    let mut dep_values: Vec<&Value> = Vec::new();\n-    for selector in &self.selector.clause {\n-      let dep_node =\n-        Node::create(\n-          selector.clone(),\n-          self.subject.clone(),\n-          self.variants.clone()\n-        );\n-      match context.get(&dep_node) {\n-        Some(&Complete::Return(ref value)) =>\n-          dep_values.push(&value),\n-        Some(&Complete::Noop(_, _)) =>\n-          return State::Complete(\n-            Complete::Noop(\"Was missing (at least) input {}.\", Some(dep_node))\n-          ),\n-        Some(&Complete::Throw(ref msg)) =>\n-          // NB: propagate thrown exception directly.\n-          return State::Complete(Complete::Throw(context.clone_val(msg))),\n-        None =>\n-          dependencies.push(dep_node),\n-      }\n-    }\n+  fn step(&self, context: Context) -> StepFuture {\n+    let deps =\n+      future::join_all(\n+        self.selector.clause.iter()\n+          .map(|selector| {\n+            context.get(&Node::create(selector.clone(), self.subject.clone(), self.variants.clone()))\n+          })\n+          .collect::<Vec<_>>()\n+      );\n \n-    if !dependencies.is_empty() {\n-      // A clause was still waiting on dependencies.\n-      State::Waiting(dependencies)\n-    } else {\n-      // Ready to run!\n-      State::Runnable(Runnable {\n-        func: self.selector.func,\n-        args: dep_values.iter().map(|v| context.clone_val(v)).collect(),\n-        cacheable: self.selector.cacheable,\n+    let selector = self.selector.clone();\n+    deps\n+      .then(move |deps_result| {\n+        match deps_result {\n+          Ok(deps) =>\n+            context.invoke_runnable(\n+              Runnable {\n+                func: selector.func,\n+                args: deps.iter().map(|v| context.clone_val(v)).collect(),\n+                cacheable: selector.cacheable,\n+              }\n+            ),\n+          Err(err) =>\n+            Err(context.was_optional(err, \"Missing at least one input.\")),\n+        }\n       })\n-    }\n+      .boxed()","path":"src/rust/engine/src/nodes.rs","position":689,"original_position":689,"commit_id":"5663dd080bb7b40f96d56019add2ee5eac755477","original_commit_id":"5663dd080bb7b40f96d56019add2ee5eac755477","user":{"login":"ity","id":1278541,"avatar_url":"https://avatars.githubusercontent.com/u/1278541?v=3","gravatar_id":"","url":"https://api.github.com/users/ity","html_url":"https://github.com/ity","followers_url":"https://api.github.com/users/ity/followers","following_url":"https://api.github.com/users/ity/following{/other_user}","gists_url":"https://api.github.com/users/ity/gists{/gist_id}","starred_url":"https://api.github.com/users/ity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ity/subscriptions","organizations_url":"https://api.github.com/users/ity/orgs","repos_url":"https://api.github.com/users/ity/repos","events_url":"https://api.github.com/users/ity/events{/privacy}","received_events_url":"https://api.github.com/users/ity/received_events","type":"User","site_admin":false},"body":"was reading through this syntax and came across https://github.com/alexcrichton/futures-rs/issues/228","created_at":"2017-02-03T14:22:14Z","updated_at":"2017-02-03T14:22:14Z","html_url":"https://github.com/pantsbuild/pants/pull/4221#discussion_r99345256","pull_request_url":"https://api.github.com/repos/pantsbuild/pants/pulls/4221","_links":{"self":{"href":"https://api.github.com/repos/pantsbuild/pants/pulls/comments/99345256"},"html":{"href":"https://github.com/pantsbuild/pants/pull/4221#discussion_r99345256"},"pull_request":{"href":"https://api.github.com/repos/pantsbuild/pants/pulls/4221"}}},"pull_request":{"url":"https://api.github.com/repos/pantsbuild/pants/pulls/4221","id":103785187,"html_url":"https://github.com/pantsbuild/pants/pull/4221","diff_url":"https://github.com/pantsbuild/pants/pull/4221.diff","patch_url":"https://github.com/pantsbuild/pants/pull/4221.patch","issue_url":"https://api.github.com/repos/pantsbuild/pants/issues/4221","number":4221,"state":"open","locked":false,"title":"[engine] Use futures for scheduling","user":{"login":"stuhood","id":46740,"avatar_url":"https://avatars.githubusercontent.com/u/46740?v=3","gravatar_id":"","url":"https://api.github.com/users/stuhood","html_url":"https://github.com/stuhood","followers_url":"https://api.github.com/users/stuhood/followers","following_url":"https://api.github.com/users/stuhood/following{/other_user}","gists_url":"https://api.github.com/users/stuhood/gists{/gist_id}","starred_url":"https://api.github.com/users/stuhood/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/stuhood/subscriptions","organizations_url":"https://api.github.com/users/stuhood/orgs","repos_url":"https://api.github.com/users/stuhood/repos","events_url":"https://api.github.com/users/stuhood/events{/privacy}","received_events_url":"https://api.github.com/users/stuhood/received_events","type":"User","site_admin":false},"body":"### Problem\r\n\r\nWhile working on #4172, a new Node type that attempted to declare large numbers of dependencies recursively was hitting the `n^2` behaviour of starting then stopping `n` times. My first attempt to fix this problem was #4214, but unfortunately that only worked in cases where dependencies had already been computed, which was insufficient in a case that was recursing through the entire filesystem.\r\n\r\nThe `Node` step model is vaguely CPS, and we've said for a while that translating it to Futures would be useful to clean it up.\r\n\r\n### Solution\r\n\r\nThe native engine now uses [futures](https://github.com/alexcrichton/futures-rs) to execute all work. This review replaces/incorporates #4214.\r\n\r\n### Result\r\n\r\nThis review regresses very slightly in terms of performance (5-10%), but unblocks fully concurrent IO on #4172, which gets us within 2x of v1 for cold runtime.","created_at":"2017-01-30T17:43:55Z","updated_at":"2017-02-03T14:22:14Z","closed_at":null,"merged_at":null,"merge_commit_sha":"a72ba89e6541ffacdf5408cafb906b1df5926591","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/pantsbuild/pants/pulls/4221/commits","review_comments_url":"https://api.github.com/repos/pantsbuild/pants/pulls/4221/comments","review_comment_url":"https://api.github.com/repos/pantsbuild/pants/pulls/comments{/number}","comments_url":"https://api.github.com/repos/pantsbuild/pants/issues/4221/comments","statuses_url":"https://api.github.com/repos/pantsbuild/pants/statuses/5663dd080bb7b40f96d56019add2ee5eac755477","head":{"label":"twitter:stuhood/engine-futures","ref":"stuhood/engine-futures","sha":"5663dd080bb7b40f96d56019add2ee5eac755477","user":{"login":"twitter","id":50278,"avatar_url":"https://avatars.githubusercontent.com/u/50278?v=3","gravatar_id":"","url":"https://api.github.com/users/twitter","html_url":"https://github.com/twitter","followers_url":"https://api.github.com/users/twitter/followers","following_url":"https://api.github.com/users/twitter/following{/other_user}","gists_url":"https://api.github.com/users/twitter/gists{/gist_id}","starred_url":"https://api.github.com/users/twitter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/twitter/subscriptions","organizations_url":"https://api.github.com/users/twitter/orgs","repos_url":"https://api.github.com/users/twitter/repos","events_url":"https://api.github.com/users/twitter/events{/privacy}","received_events_url":"https://api.github.com/users/twitter/received_events","type":"Organization","site_admin":false},"repo":{"id":29317658,"name":"pants","full_name":"twitter/pants","owner":{"login":"twitter","id":50278,"avatar_url":"https://avatars.githubusercontent.com/u/50278?v=3","gravatar_id":"","url":"https://api.github.com/users/twitter","html_url":"https://github.com/twitter","followers_url":"https://api.github.com/users/twitter/followers","following_url":"https://api.github.com/users/twitter/following{/other_user}","gists_url":"https://api.github.com/users/twitter/gists{/gist_id}","starred_url":"https://api.github.com/users/twitter/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/twitter/subscriptions","organizations_url":"https://api.github.com/users/twitter/orgs","repos_url":"https://api.github.com/users/twitter/repos","events_url":"https://api.github.com/users/twitter/events{/privacy}","received_events_url":"https://api.github.com/users/twitter/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/twitter/pants","description":"Pants Build System","fork":true,"url":"https://api.github.com/repos/twitter/pants","forks_url":"https://api.github.com/repos/twitter/pants/forks","keys_url":"https://api.github.com/repos/twitter/pants/keys{/key_id}","collaborators_url":"https://api.github.com/repos/twitter/pants/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/twitter/pants/teams","hooks_url":"https://api.github.com/repos/twitter/pants/hooks","issue_events_url":"https://api.github.com/repos/twitter/pants/issues/events{/number}","events_url":"https://api.github.com/repos/twitter/pants/events","assignees_url":"https://api.github.com/repos/twitter/pants/assignees{/user}","branches_url":"https://api.github.com/repos/twitter/pants/branches{/branch}","tags_url":"https://api.github.com/repos/twitter/pants/tags","blobs_url":"https://api.github.com/repos/twitter/pants/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/twitter/pants/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/twitter/pants/git/refs{/sha}","trees_url":"https://api.github.com/repos/twitter/pants/git/trees{/sha}","statuses_url":"https://api.github.com/repos/twitter/pants/statuses/{sha}","languages_url":"https://api.github.com/repos/twitter/pants/languages","stargazers_url":"https://api.github.com/repos/twitter/pants/stargazers","contributors_url":"https://api.github.com/repos/twitter/pants/contributors","subscribers_url":"https://api.github.com/repos/twitter/pants/subscribers","subscription_url":"https://api.github.com/repos/twitter/pants/subscription","commits_url":"https://api.github.com/repos/twitter/pants/commits{/sha}","git_commits_url":"https://api.github.com/repos/twitter/pants/git/commits{/sha}","comments_url":"https://api.github.com/repos/twitter/pants/comments{/number}","issue_comment_url":"https://api.github.com/repos/twitter/pants/issues/comments{/number}","contents_url":"https://api.github.com/repos/twitter/pants/contents/{+path}","compare_url":"https://api.github.com/repos/twitter/pants/compare/{base}...{head}","merges_url":"https://api.github.com/repos/twitter/pants/merges","archive_url":"https://api.github.com/repos/twitter/pants/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/twitter/pants/downloads","issues_url":"https://api.github.com/repos/twitter/pants/issues{/number}","pulls_url":"https://api.github.com/repos/twitter/pants/pulls{/number}","milestones_url":"https://api.github.com/repos/twitter/pants/milestones{/number}","notifications_url":"https://api.github.com/repos/twitter/pants/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/twitter/pants/labels{/name}","releases_url":"https://api.github.com/repos/twitter/pants/releases{/id}","deployments_url":"https://api.github.com/repos/twitter/pants/deployments","created_at":"2015-01-15T20:51:58Z","updated_at":"2017-01-13T23:38:40Z","pushed_at":"2017-02-03T07:05:15Z","git_url":"git://github.com/twitter/pants.git","ssh_url":"git@github.com:twitter/pants.git","clone_url":"https://github.com/twitter/pants.git","svn_url":"https://github.com/twitter/pants","homepage":"http://pantsbuild.github.io/","size":81032,"stargazers_count":39,"watchers_count":39,"language":"Python","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":12,"mirror_url":null,"open_issues_count":0,"forks":12,"open_issues":0,"watchers":39,"default_branch":"master"}},"base":{"label":"pantsbuild:master","ref":"master","sha":"3641dd30c1dd9c5468175b6ce25b65424e4c2704","user":{"login":"pantsbuild","id":3065172,"avatar_url":"https://avatars.githubusercontent.com/u/3065172?v=3","gravatar_id":"","url":"https://api.github.com/users/pantsbuild","html_url":"https://github.com/pantsbuild","followers_url":"https://api.github.com/users/pantsbuild/followers","following_url":"https://api.github.com/users/pantsbuild/following{/other_user}","gists_url":"https://api.github.com/users/pantsbuild/gists{/gist_id}","starred_url":"https://api.github.com/users/pantsbuild/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pantsbuild/subscriptions","organizations_url":"https://api.github.com/users/pantsbuild/orgs","repos_url":"https://api.github.com/users/pantsbuild/repos","events_url":"https://api.github.com/users/pantsbuild/events{/privacy}","received_events_url":"https://api.github.com/users/pantsbuild/received_events","type":"Organization","site_admin":false},"repo":{"id":7209075,"name":"pants","full_name":"pantsbuild/pants","owner":{"login":"pantsbuild","id":3065172,"avatar_url":"https://avatars.githubusercontent.com/u/3065172?v=3","gravatar_id":"","url":"https://api.github.com/users/pantsbuild","html_url":"https://github.com/pantsbuild","followers_url":"https://api.github.com/users/pantsbuild/followers","following_url":"https://api.github.com/users/pantsbuild/following{/other_user}","gists_url":"https://api.github.com/users/pantsbuild/gists{/gist_id}","starred_url":"https://api.github.com/users/pantsbuild/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pantsbuild/subscriptions","organizations_url":"https://api.github.com/users/pantsbuild/orgs","repos_url":"https://api.github.com/users/pantsbuild/repos","events_url":"https://api.github.com/users/pantsbuild/events{/privacy}","received_events_url":"https://api.github.com/users/pantsbuild/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/pantsbuild/pants","description":"Pants Build System","fork":false,"url":"https://api.github.com/repos/pantsbuild/pants","forks_url":"https://api.github.com/repos/pantsbuild/pants/forks","keys_url":"https://api.github.com/repos/pantsbuild/pants/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pantsbuild/pants/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pantsbuild/pants/teams","hooks_url":"https://api.github.com/repos/pantsbuild/pants/hooks","issue_events_url":"https://api.github.com/repos/pantsbuild/pants/issues/events{/number}","events_url":"https://api.github.com/repos/pantsbuild/pants/events","assignees_url":"https://api.github.com/repos/pantsbuild/pants/assignees{/user}","branches_url":"https://api.github.com/repos/pantsbuild/pants/branches{/branch}","tags_url":"https://api.github.com/repos/pantsbuild/pants/tags","blobs_url":"https://api.github.com/repos/pantsbuild/pants/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pantsbuild/pants/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pantsbuild/pants/git/refs{/sha}","trees_url":"https://api.github.com/repos/pantsbuild/pants/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pantsbuild/pants/statuses/{sha}","languages_url":"https://api.github.com/repos/pantsbuild/pants/languages","stargazers_url":"https://api.github.com/repos/pantsbuild/pants/stargazers","contributors_url":"https://api.github.com/repos/pantsbuild/pants/contributors","subscribers_url":"https://api.github.com/repos/pantsbuild/pants/subscribers","subscription_url":"https://api.github.com/repos/pantsbuild/pants/subscription","commits_url":"https://api.github.com/repos/pantsbuild/pants/commits{/sha}","git_commits_url":"https://api.github.com/repos/pantsbuild/pants/git/commits{/sha}","comments_url":"https://api.github.com/repos/pantsbuild/pants/comments{/number}","issue_comment_url":"https://api.github.com/repos/pantsbuild/pants/issues/comments{/number}","contents_url":"https://api.github.com/repos/pantsbuild/pants/contents/{+path}","compare_url":"https://api.github.com/repos/pantsbuild/pants/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pantsbuild/pants/merges","archive_url":"https://api.github.com/repos/pantsbuild/pants/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pantsbuild/pants/downloads","issues_url":"https://api.github.com/repos/pantsbuild/pants/issues{/number}","pulls_url":"https://api.github.com/repos/pantsbuild/pants/pulls{/number}","milestones_url":"https://api.github.com/repos/pantsbuild/pants/milestones{/number}","notifications_url":"https://api.github.com/repos/pantsbuild/pants/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pantsbuild/pants/labels{/name}","releases_url":"https://api.github.com/repos/pantsbuild/pants/releases{/id}","deployments_url":"https://api.github.com/repos/pantsbuild/pants/deployments","created_at":"2012-12-17T17:39:04Z","updated_at":"2017-01-31T05:38:02Z","pushed_at":"2017-02-03T12:36:15Z","git_url":"git://github.com/pantsbuild/pants.git","ssh_url":"git@github.com:pantsbuild/pants.git","clone_url":"https://github.com/pantsbuild/pants.git","svn_url":"https://github.com/pantsbuild/pants","homepage":"http://pantsbuild.github.io/","size":78212,"stargazers_count":723,"watchers_count":723,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":212,"mirror_url":null,"open_issues_count":459,"forks":212,"open_issues":459,"watchers":723,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/pantsbuild/pants/pulls/4221"},"html":{"href":"https://github.com/pantsbuild/pants/pull/4221"},"issue":{"href":"https://api.github.com/repos/pantsbuild/pants/issues/4221"},"comments":{"href":"https://api.github.com/repos/pantsbuild/pants/issues/4221/comments"},"review_comments":{"href":"https://api.github.com/repos/pantsbuild/pants/pulls/4221/comments"},"review_comment":{"href":"https://api.github.com/repos/pantsbuild/pants/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/pantsbuild/pants/pulls/4221/commits"},"statuses":{"href":"https://api.github.com/repos/pantsbuild/pants/statuses/5663dd080bb7b40f96d56019add2ee5eac755477"}}}}
{"actor":{"display_login":"ity"}}