{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/19317","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/19317/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/19317/comments","events_url":"https://api.github.com/repos/apache/spark/issues/19317/events","html_url":"https://github.com/apache/spark/pull/19317","id":259697310,"number":19317,"title":"[SPARK-22098][CORE] Add new method aggregateByKeyLocally in RDD","user":{"login":"ConeyLiu","id":12733256,"avatar_url":"https://avatars3.githubusercontent.com/u/12733256?v=4","gravatar_id":"","url":"https://api.github.com/users/ConeyLiu","html_url":"https://github.com/ConeyLiu","followers_url":"https://api.github.com/users/ConeyLiu/followers","following_url":"https://api.github.com/users/ConeyLiu/following{/other_user}","gists_url":"https://api.github.com/users/ConeyLiu/gists{/gist_id}","starred_url":"https://api.github.com/users/ConeyLiu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ConeyLiu/subscriptions","organizations_url":"https://api.github.com/users/ConeyLiu/orgs","repos_url":"https://api.github.com/users/ConeyLiu/repos","events_url":"https://api.github.com/users/ConeyLiu/events{/privacy}","received_events_url":"https://api.github.com/users/ConeyLiu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-09-22T04:05:34Z","updated_at":"2017-09-22T18:12:50Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/19317","html_url":"https://github.com/apache/spark/pull/19317","diff_url":"https://github.com/apache/spark/pull/19317.diff","patch_url":"https://github.com/apache/spark/pull/19317.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nhttps://issues.apache.org/jira/browse/SPARK-22096\r\n\r\nNaiveBayes currently takes aggreateByKey followed by a collect to calculate frequency for each feature/label. We can implement a new function 'aggregateByKeyLocally' in RDD that merges locally on each mapper before sending results to a reducer to save one stage.\r\nWe tested on NaiveBayes and see ~20% performance gain with these changes.\r\n\r\nThis is a subtask of our improvement.\r\n\r\n## How was this patch tested?\r\n\r\nNew UT.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/331521734","html_url":"https://github.com/apache/spark/pull/19317#issuecomment-331521734","issue_url":"https://api.github.com/repos/apache/spark/issues/19317","id":331521734,"user":{"login":"jiangxb1987","id":4784782,"avatar_url":"https://avatars1.githubusercontent.com/u/4784782?v=4","gravatar_id":"","url":"https://api.github.com/users/jiangxb1987","html_url":"https://github.com/jiangxb1987","followers_url":"https://api.github.com/users/jiangxb1987/followers","following_url":"https://api.github.com/users/jiangxb1987/following{/other_user}","gists_url":"https://api.github.com/users/jiangxb1987/gists{/gist_id}","starred_url":"https://api.github.com/users/jiangxb1987/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jiangxb1987/subscriptions","organizations_url":"https://api.github.com/users/jiangxb1987/orgs","repos_url":"https://api.github.com/users/jiangxb1987/repos","events_url":"https://api.github.com/users/jiangxb1987/events{/privacy}","received_events_url":"https://api.github.com/users/jiangxb1987/received_events","type":"User","site_admin":false},"created_at":"2017-09-22T18:12:50Z","updated_at":"2017-09-22T18:12:50Z","author_association":"CONTRIBUTOR","body":"cc @WeichenXu123 mind take a look?"}}
{"actor":{"display_login":"jiangxb1987"}}