{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/18162","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/18162/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/18162/comments","events_url":"https://api.github.com/repos/apache/spark/issues/18162/events","html_url":"https://github.com/apache/spark/pull/18162","id":232590284,"number":18162,"title":"[SPARK-20923] turn tracking of TaskMetrics._updatedBlockStatuses off","user":{"login":"tgravescs","id":4563792,"avatar_url":"https://avatars1.githubusercontent.com/u/4563792?v=3","gravatar_id":"","url":"https://api.github.com/users/tgravescs","html_url":"https://github.com/tgravescs","followers_url":"https://api.github.com/users/tgravescs/followers","following_url":"https://api.github.com/users/tgravescs/following{/other_user}","gists_url":"https://api.github.com/users/tgravescs/gists{/gist_id}","starred_url":"https://api.github.com/users/tgravescs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgravescs/subscriptions","organizations_url":"https://api.github.com/users/tgravescs/orgs","repos_url":"https://api.github.com/users/tgravescs/repos","events_url":"https://api.github.com/users/tgravescs/events{/privacy}","received_events_url":"https://api.github.com/users/tgravescs/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":33,"created_at":"2017-05-31T14:46:47Z","updated_at":"2017-06-22T18:02:42Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/18162","html_url":"https://github.com/apache/spark/pull/18162","diff_url":"https://github.com/apache/spark/pull/18162.diff","patch_url":"https://github.com/apache/spark/pull/18162.patch"},"body":"## What changes were proposed in this pull request?\r\nTurn tracking of TaskMetrics._updatedBlockStatuses off by default. As far as I can see its not used by anything and it uses a lot of memory when caching and processing a lot of blocks.  In my case it was taking 5GB of a 10GB heap and I even went up to 50GB heap and the job still ran out of memory.  With this change in place the same job easily runs in less then 10GB of heap.\r\n\r\nWe leave the api there as well as a config to turn it back on just in case anyone is using it.  TaskMetrics is exposed via SparkListenerTaskEnd so if users are relying on it they can turn it back on.\r\n\r\n## How was this patch tested?\r\n\r\nRan unit tests that were modified and manually tested on a couple of jobs (with and without caching).  Clicked through the UI and didn't see anything missing. \r\nRan my very large hive query job with 200,000 small tasks, 1000 executors, cached 6+TB of data this runs fine now whereas without this change it would go into full gcs and eventually die."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/310457028","html_url":"https://github.com/apache/spark/pull/18162#issuecomment-310457028","issue_url":"https://api.github.com/repos/apache/spark/issues/18162","id":310457028,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars2.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2017-06-22T18:02:42Z","updated_at":"2017-06-22T18:02:42Z","body":"**[Test build #78467 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/78467/testReport)** for PR 18162 at commit [`68fbd9f`](https://github.com/apache/spark/commit/68fbd9f8c1c2981eff859b1fec078e238aa0280c)."}}
{"actor":{"display_login":"SparkQA"}}