{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/18679","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/18679/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/18679/comments","events_url":"https://api.github.com/repos/apache/spark/issues/18679/events","html_url":"https://github.com/apache/spark/pull/18679","id":243962048,"number":18679,"title":"[SPARK-21319][SQL] Fix memory leak in sorter","user":{"login":"cloud-fan","id":3182036,"avatar_url":"https://avatars3.githubusercontent.com/u/3182036?v=4","gravatar_id":"","url":"https://api.github.com/users/cloud-fan","html_url":"https://github.com/cloud-fan","followers_url":"https://api.github.com/users/cloud-fan/followers","following_url":"https://api.github.com/users/cloud-fan/following{/other_user}","gists_url":"https://api.github.com/users/cloud-fan/gists{/gist_id}","starred_url":"https://api.github.com/users/cloud-fan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cloud-fan/subscriptions","organizations_url":"https://api.github.com/users/cloud-fan/orgs","repos_url":"https://api.github.com/users/cloud-fan/repos","events_url":"https://api.github.com/users/cloud-fan/events{/privacy}","received_events_url":"https://api.github.com/users/cloud-fan/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-07-19T08:30:19Z","updated_at":"2017-07-19T08:32:26Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/18679","html_url":"https://github.com/apache/spark/pull/18679","diff_url":"https://github.com/apache/spark/pull/18679.diff","patch_url":"https://github.com/apache/spark/pull/18679.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\n`UnsafeExternalSorter.recordComparator` can be either `KVComparator` or `RowComparator`, and both of them will keep the reference to the input rows they compared last time.\r\n\r\nAfter sorting, we return the sorted iterator to upstream operators. However, the upstream operators may take a while to consume up the sorted iterator, and `UnsafeExternalSorter` is registered to `TaskContext` at [here](https://github.com/apache/spark/blob/master/core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java#L159-L161), which means we will keep the `UnsafeExternalSorter` instance and keep the last compared input rows in memory until the sorted iterator is consumed up.\r\n\r\nThings get worse if we sort within partitions of a dataset and coalesce all partitions into one, as we will keep a lot of input rows in memory and the time to consume up all the sorted iterators is long.\r\n\r\nThis PR propose to remove the reference of the input rows in the comparator after the sort is finished.\r\n\r\n## How was this patch tested?\r\n\r\n(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)\r\n(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)\r\n\r\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/316313249","html_url":"https://github.com/apache/spark/pull/18679#issuecomment-316313249","issue_url":"https://api.github.com/repos/apache/spark/issues/18679","id":316313249,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2017-07-19T08:32:26Z","updated_at":"2017-07-19T08:32:26Z","body":"**[Test build #79746 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79746/testReport)** for PR 18679 at commit [`f5752d4`](https://github.com/apache/spark/commit/f5752d4a1ed5b6dc1b1d2dea4056ff8ebb65f657)."}}
{"actor":{"display_login":"SparkQA"}}