{"action":"created","comment":{"url":"https://api.github.com/repos/apache/spark/pulls/comments/101598818","pull_request_review_id":22333559,"id":101598818,"diff_hunk":"@@ -365,17 +385,66 @@ object TypeCoercion {\n   }\n \n   /**\n-   * Convert the value and in list expressions to the common operator type\n-   * by looking at all the argument types and finding the closest one that\n-   * all the arguments can be cast to. When no common operator type is found\n-   * the original expression will be returned and an Analysis Exception will\n-   * be raised at type checking phase.\n+   * Handles type coercion for both IN expression with subquery and IN\n+   * expressions without subquery.\n+   * 1. In the first case, find the common type by comparing the left hand side\n+   *    expression types against corresponding right hand side expression derived\n+   *    from the subquery expression's plan output. Inject appropriate casts in the\n+   *    LHS and RHS side of IN expression.\n+   *\n+   * 2. In the second case, convert the value and in list expressions to the\n+   *    common operator type by looking at all the argument types and finding\n+   *    the closest one that all the arguments can be cast to. When no common\n+   *    operator type is found the original expression will be returned and an\n+   *    Analysis Exception will be raised at the type checking phase.\n    */\n   object InConversion extends Rule[LogicalPlan] {\n     def apply(plan: LogicalPlan): LogicalPlan = plan resolveExpressions {\n       // Skip nodes who's children have not been resolved yet.\n       case e if !e.childrenResolved => e\n \n+      // Handle type casting required between value expression and subquery output\n+      // in IN subquery.\n+      case i @ In(a, Seq(ListQuery(sub, children, exprId))) if !i.resolved =>","path":"sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/TypeCoercion.scala","position":56,"original_position":56,"commit_id":"33844bb8c6e72c35e12a7ea9f124ad3c6b8bd43c","original_commit_id":"33844bb8c6e72c35e12a7ea9f124ad3c6b8bd43c","user":{"login":"dilipbiswal","id":14225158,"avatar_url":"https://avatars.githubusercontent.com/u/14225158?v=3","gravatar_id":"","url":"https://api.github.com/users/dilipbiswal","html_url":"https://github.com/dilipbiswal","followers_url":"https://api.github.com/users/dilipbiswal/followers","following_url":"https://api.github.com/users/dilipbiswal/following{/other_user}","gists_url":"https://api.github.com/users/dilipbiswal/gists{/gist_id}","starred_url":"https://api.github.com/users/dilipbiswal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dilipbiswal/subscriptions","organizations_url":"https://api.github.com/users/dilipbiswal/orgs","repos_url":"https://api.github.com/users/dilipbiswal/repos","events_url":"https://api.github.com/users/dilipbiswal/events{/privacy}","received_events_url":"https://api.github.com/users/dilipbiswal/received_events","type":"User","site_admin":false},"body":"Plan example is available in [Code Changes](https://docs.google.com/document/d/18mqjhL9V1An-tNta7aVE13HkALRZ5GZ24AATA-Vqqf0/edit#) in section 3.","created_at":"2017-02-16T18:58:29Z","updated_at":"2017-02-16T18:58:29Z","html_url":"https://github.com/apache/spark/pull/16954#discussion_r101598818","pull_request_url":"https://api.github.com/repos/apache/spark/pulls/16954","_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/comments/101598818"},"html":{"href":"https://github.com/apache/spark/pull/16954#discussion_r101598818"},"pull_request":{"href":"https://api.github.com/repos/apache/spark/pulls/16954"}}},"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16954","id":106479943,"html_url":"https://github.com/apache/spark/pull/16954","diff_url":"https://github.com/apache/spark/pull/16954.diff","patch_url":"https://github.com/apache/spark/pull/16954.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/16954","number":16954,"state":"open","locked":false,"title":"[SPARK-18874][SQL] First phase: Deferring the correlated predicate pull up to Optimizer phase","user":{"login":"dilipbiswal","id":14225158,"avatar_url":"https://avatars.githubusercontent.com/u/14225158?v=3","gravatar_id":"","url":"https://api.github.com/users/dilipbiswal","html_url":"https://github.com/dilipbiswal","followers_url":"https://api.github.com/users/dilipbiswal/followers","following_url":"https://api.github.com/users/dilipbiswal/following{/other_user}","gists_url":"https://api.github.com/users/dilipbiswal/gists{/gist_id}","starred_url":"https://api.github.com/users/dilipbiswal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dilipbiswal/subscriptions","organizations_url":"https://api.github.com/users/dilipbiswal/orgs","repos_url":"https://api.github.com/users/dilipbiswal/repos","events_url":"https://api.github.com/users/dilipbiswal/events{/privacy}","received_events_url":"https://api.github.com/users/dilipbiswal/received_events","type":"User","site_admin":false},"body":"## What changes were proposed in this pull request?\r\nCurrently Analyzer as part of ResolveSubquery, pulls up the correlated predicates to its\r\noriginating SubqueryExpression. The subquery plan is then transformed to remove the correlated\r\npredicates after they are moved up to the outer plan. In this PR, the task of pulling up\r\ncorrelated predicates is deferred to Optimizer. This is the initial work that will allow us to\r\nsupport the form of correlated subqueries that we don't support today. The design document\r\nfrom @nsyca can be found in the following link :\r\n[DesignDoc](https://docs.google.com/document/d/1QDZ8JwU63RwGFS6KVF54Rjj9ZJyK33d49ZWbjFBaIgU/edit#)\r\n\r\nThe brief description of code changes (hopefully to aid with code review) can be be found in the\r\nfollowing link:\r\n[CodeChanges](https://docs.google.com/document/d/18mqjhL9V1An-tNta7aVE13HkALRZ5GZ24AATA-Vqqf0/edit#)\r\n\r\n\r\n## How was this patch tested?\r\nThe test case PRs were submitted earlier using.\r\n[16337](https://github.com/apache/spark/pull/16337) [16759](https://github.com/apache/spark/pull/16759) [16841](https://github.com/apache/spark/pull/16841) [16915](https://github.com/apache/spark/pull/16915) [16798](https://github.com/apache/spark/pull/16798) [16712](https://github.com/apache/spark/pull/16712) [16710](https://github.com/apache/spark/pull/16710) [16760](https://github.com/apache/spark/pull/16760) [16802](https://github.com/apache/spark/pull/16802)\r\n","created_at":"2017-02-16T08:58:29Z","updated_at":"2017-02-16T18:58:29Z","closed_at":null,"merged_at":null,"merge_commit_sha":"ab44c99179300263af6a26d56a9aae102fa78b34","assignee":null,"assignees":[],"milestone":null,"commits_url":"https://api.github.com/repos/apache/spark/pulls/16954/commits","review_comments_url":"https://api.github.com/repos/apache/spark/pulls/16954/comments","review_comment_url":"https://api.github.com/repos/apache/spark/pulls/comments{/number}","comments_url":"https://api.github.com/repos/apache/spark/issues/16954/comments","statuses_url":"https://api.github.com/repos/apache/spark/statuses/33844bb8c6e72c35e12a7ea9f124ad3c6b8bd43c","head":{"label":"dilipbiswal:SPARK-18874","ref":"SPARK-18874","sha":"33844bb8c6e72c35e12a7ea9f124ad3c6b8bd43c","user":{"login":"dilipbiswal","id":14225158,"avatar_url":"https://avatars.githubusercontent.com/u/14225158?v=3","gravatar_id":"","url":"https://api.github.com/users/dilipbiswal","html_url":"https://github.com/dilipbiswal","followers_url":"https://api.github.com/users/dilipbiswal/followers","following_url":"https://api.github.com/users/dilipbiswal/following{/other_user}","gists_url":"https://api.github.com/users/dilipbiswal/gists{/gist_id}","starred_url":"https://api.github.com/users/dilipbiswal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dilipbiswal/subscriptions","organizations_url":"https://api.github.com/users/dilipbiswal/orgs","repos_url":"https://api.github.com/users/dilipbiswal/repos","events_url":"https://api.github.com/users/dilipbiswal/events{/privacy}","received_events_url":"https://api.github.com/users/dilipbiswal/received_events","type":"User","site_admin":false},"repo":{"id":42921897,"name":"spark","full_name":"dilipbiswal/spark","owner":{"login":"dilipbiswal","id":14225158,"avatar_url":"https://avatars.githubusercontent.com/u/14225158?v=3","gravatar_id":"","url":"https://api.github.com/users/dilipbiswal","html_url":"https://github.com/dilipbiswal","followers_url":"https://api.github.com/users/dilipbiswal/followers","following_url":"https://api.github.com/users/dilipbiswal/following{/other_user}","gists_url":"https://api.github.com/users/dilipbiswal/gists{/gist_id}","starred_url":"https://api.github.com/users/dilipbiswal/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dilipbiswal/subscriptions","organizations_url":"https://api.github.com/users/dilipbiswal/orgs","repos_url":"https://api.github.com/users/dilipbiswal/repos","events_url":"https://api.github.com/users/dilipbiswal/events{/privacy}","received_events_url":"https://api.github.com/users/dilipbiswal/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/dilipbiswal/spark","description":"Mirror of Apache Spark","fork":true,"url":"https://api.github.com/repos/dilipbiswal/spark","forks_url":"https://api.github.com/repos/dilipbiswal/spark/forks","keys_url":"https://api.github.com/repos/dilipbiswal/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/dilipbiswal/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/dilipbiswal/spark/teams","hooks_url":"https://api.github.com/repos/dilipbiswal/spark/hooks","issue_events_url":"https://api.github.com/repos/dilipbiswal/spark/issues/events{/number}","events_url":"https://api.github.com/repos/dilipbiswal/spark/events","assignees_url":"https://api.github.com/repos/dilipbiswal/spark/assignees{/user}","branches_url":"https://api.github.com/repos/dilipbiswal/spark/branches{/branch}","tags_url":"https://api.github.com/repos/dilipbiswal/spark/tags","blobs_url":"https://api.github.com/repos/dilipbiswal/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/dilipbiswal/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/dilipbiswal/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/dilipbiswal/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/dilipbiswal/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/dilipbiswal/spark/languages","stargazers_url":"https://api.github.com/repos/dilipbiswal/spark/stargazers","contributors_url":"https://api.github.com/repos/dilipbiswal/spark/contributors","subscribers_url":"https://api.github.com/repos/dilipbiswal/spark/subscribers","subscription_url":"https://api.github.com/repos/dilipbiswal/spark/subscription","commits_url":"https://api.github.com/repos/dilipbiswal/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/dilipbiswal/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/dilipbiswal/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/dilipbiswal/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/dilipbiswal/spark/contents/{+path}","compare_url":"https://api.github.com/repos/dilipbiswal/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/dilipbiswal/spark/merges","archive_url":"https://api.github.com/repos/dilipbiswal/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/dilipbiswal/spark/downloads","issues_url":"https://api.github.com/repos/dilipbiswal/spark/issues{/number}","pulls_url":"https://api.github.com/repos/dilipbiswal/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/dilipbiswal/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/dilipbiswal/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/dilipbiswal/spark/labels{/name}","releases_url":"https://api.github.com/repos/dilipbiswal/spark/releases{/id}","deployments_url":"https://api.github.com/repos/dilipbiswal/spark/deployments","created_at":"2015-09-22T08:39:46Z","updated_at":"2016-04-06T08:56:13Z","pushed_at":"2017-02-16T07:44:19Z","git_url":"git://github.com/dilipbiswal/spark.git","ssh_url":"git@github.com:dilipbiswal/spark.git","clone_url":"https://github.com/dilipbiswal/spark.git","svn_url":"https://github.com/dilipbiswal/spark","homepage":null,"size":251929,"stargazers_count":0,"watchers_count":0,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"apache:master","ref":"master","sha":"3871d94a695d47169720e877f77ff1e4bede43ee","user":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"repo":{"id":17165658,"name":"spark","full_name":"apache/spark","owner":{"login":"apache","id":47359,"avatar_url":"https://avatars.githubusercontent.com/u/47359?v=3","gravatar_id":"","url":"https://api.github.com/users/apache","html_url":"https://github.com/apache","followers_url":"https://api.github.com/users/apache/followers","following_url":"https://api.github.com/users/apache/following{/other_user}","gists_url":"https://api.github.com/users/apache/gists{/gist_id}","starred_url":"https://api.github.com/users/apache/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/apache/subscriptions","organizations_url":"https://api.github.com/users/apache/orgs","repos_url":"https://api.github.com/users/apache/repos","events_url":"https://api.github.com/users/apache/events{/privacy}","received_events_url":"https://api.github.com/users/apache/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/apache/spark","description":"Mirror of Apache Spark","fork":false,"url":"https://api.github.com/repos/apache/spark","forks_url":"https://api.github.com/repos/apache/spark/forks","keys_url":"https://api.github.com/repos/apache/spark/keys{/key_id}","collaborators_url":"https://api.github.com/repos/apache/spark/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/apache/spark/teams","hooks_url":"https://api.github.com/repos/apache/spark/hooks","issue_events_url":"https://api.github.com/repos/apache/spark/issues/events{/number}","events_url":"https://api.github.com/repos/apache/spark/events","assignees_url":"https://api.github.com/repos/apache/spark/assignees{/user}","branches_url":"https://api.github.com/repos/apache/spark/branches{/branch}","tags_url":"https://api.github.com/repos/apache/spark/tags","blobs_url":"https://api.github.com/repos/apache/spark/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/apache/spark/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/apache/spark/git/refs{/sha}","trees_url":"https://api.github.com/repos/apache/spark/git/trees{/sha}","statuses_url":"https://api.github.com/repos/apache/spark/statuses/{sha}","languages_url":"https://api.github.com/repos/apache/spark/languages","stargazers_url":"https://api.github.com/repos/apache/spark/stargazers","contributors_url":"https://api.github.com/repos/apache/spark/contributors","subscribers_url":"https://api.github.com/repos/apache/spark/subscribers","subscription_url":"https://api.github.com/repos/apache/spark/subscription","commits_url":"https://api.github.com/repos/apache/spark/commits{/sha}","git_commits_url":"https://api.github.com/repos/apache/spark/git/commits{/sha}","comments_url":"https://api.github.com/repos/apache/spark/comments{/number}","issue_comment_url":"https://api.github.com/repos/apache/spark/issues/comments{/number}","contents_url":"https://api.github.com/repos/apache/spark/contents/{+path}","compare_url":"https://api.github.com/repos/apache/spark/compare/{base}...{head}","merges_url":"https://api.github.com/repos/apache/spark/merges","archive_url":"https://api.github.com/repos/apache/spark/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/apache/spark/downloads","issues_url":"https://api.github.com/repos/apache/spark/issues{/number}","pulls_url":"https://api.github.com/repos/apache/spark/pulls{/number}","milestones_url":"https://api.github.com/repos/apache/spark/milestones{/number}","notifications_url":"https://api.github.com/repos/apache/spark/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/apache/spark/labels{/name}","releases_url":"https://api.github.com/repos/apache/spark/releases{/id}","deployments_url":"https://api.github.com/repos/apache/spark/deployments","created_at":"2014-02-25T08:00:08Z","updated_at":"2017-02-16T17:29:26Z","pushed_at":"2017-02-16T18:50:55Z","git_url":"git://github.com/apache/spark.git","ssh_url":"git@github.com:apache/spark.git","clone_url":"https://github.com/apache/spark.git","svn_url":"https://github.com/apache/spark","homepage":null,"size":263083,"stargazers_count":11709,"watchers_count":11709,"language":"Scala","has_issues":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":10977,"mirror_url":"git://git.apache.org/spark.git","open_issues_count":473,"forks":10977,"open_issues":473,"watchers":11709,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/apache/spark/pulls/16954"},"html":{"href":"https://github.com/apache/spark/pull/16954"},"issue":{"href":"https://api.github.com/repos/apache/spark/issues/16954"},"comments":{"href":"https://api.github.com/repos/apache/spark/issues/16954/comments"},"review_comments":{"href":"https://api.github.com/repos/apache/spark/pulls/16954/comments"},"review_comment":{"href":"https://api.github.com/repos/apache/spark/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/apache/spark/pulls/16954/commits"},"statuses":{"href":"https://api.github.com/repos/apache/spark/statuses/33844bb8c6e72c35e12a7ea9f124ad3c6b8bd43c"}}}}
{"actor":{"display_login":"dilipbiswal"}}