{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/19175","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/19175/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/19175/comments","events_url":"https://api.github.com/repos/apache/spark/issues/19175/events","html_url":"https://github.com/apache/spark/pull/19175","id":256440146,"number":19175,"title":"[SPARK-21964][SQL]Enable splitting the Aggregate (on Expand) into a number of Aggregates for grouing analytics","user":{"login":"DonnyZone","id":29158321,"avatar_url":"https://avatars0.githubusercontent.com/u/29158321?v=4","gravatar_id":"","url":"https://api.github.com/users/DonnyZone","html_url":"https://github.com/DonnyZone","followers_url":"https://api.github.com/users/DonnyZone/followers","following_url":"https://api.github.com/users/DonnyZone/following{/other_user}","gists_url":"https://api.github.com/users/DonnyZone/gists{/gist_id}","starred_url":"https://api.github.com/users/DonnyZone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/DonnyZone/subscriptions","organizations_url":"https://api.github.com/users/DonnyZone/orgs","repos_url":"https://api.github.com/users/DonnyZone/repos","events_url":"https://api.github.com/users/DonnyZone/events{/privacy}","received_events_url":"https://api.github.com/users/DonnyZone/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2017-09-09T13:49:46Z","updated_at":"2017-09-26T07:05:07Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/19175","html_url":"https://github.com/apache/spark/pull/19175","diff_url":"https://github.com/apache/spark/pull/19175.diff","patch_url":"https://github.com/apache/spark/pull/19175.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nhttps://issues.apache.org/jira/browse/SPARK-21964\r\n\r\nCurrent implementation for grouping analytics (i.e., cube, rollup and grouping sets) is \"heavyweight\" for many scenarios (e.g., high dimensions cube), as the Expand operator produces a large number of projections, resulting vast shuffle write.  It may result into low performance or even OOM issues for direct buffer memory.\r\n\r\nThis PR provides another choice which enables splitting the heavyweight aggregate into a number of lightweight aggregates for each group. Actually, it implements the grouping analytics as Union and executes the aggregates one by one. \r\n\r\nThis optimization is opposite to the general sense of \"avoding redundant data scan\". However, it can achieve overall high performance for many cases. In our production environment, about 500+ queries can get benefit from it. \r\n\r\nThe current splitting strategy is simple as one aggregation for one group. In future, we may figure out more intelligent splitting stategies (e.g., cost-based method).\r\n\r\n## How was this patch tested?\r\nUnit tests\r\nManual tests in production environment\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/332106113","html_url":"https://github.com/apache/spark/pull/19175#issuecomment-332106113","issue_url":"https://api.github.com/repos/apache/spark/issues/19175","id":332106113,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2017-09-26T07:05:07Z","updated_at":"2017-09-26T07:05:07Z","author_association":"NONE","body":"Test FAILed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82181/\nTest FAILed."}}
{"actor":{"display_login":"AmplabJenkins"}}