{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/16603","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/16603/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/16603/comments","events_url":"https://api.github.com/repos/apache/spark/issues/16603/events","html_url":"https://github.com/apache/spark/pull/16603","id":201019779,"number":16603,"title":"[SPARK-19244][Core] Sort MemoryConsumers according to their memory usage when spilling","user":{"login":"viirya","id":68855,"avatar_url":"https://avatars.githubusercontent.com/u/68855?v=3","gravatar_id":"","url":"https://api.github.com/users/viirya","html_url":"https://github.com/viirya","followers_url":"https://api.github.com/users/viirya/followers","following_url":"https://api.github.com/users/viirya/following{/other_user}","gists_url":"https://api.github.com/users/viirya/gists{/gist_id}","starred_url":"https://api.github.com/users/viirya/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/viirya/subscriptions","organizations_url":"https://api.github.com/users/viirya/orgs","repos_url":"https://api.github.com/users/viirya/repos","events_url":"https://api.github.com/users/viirya/events{/privacy}","received_events_url":"https://api.github.com/users/viirya/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2017-01-16T13:32:11Z","updated_at":"2017-01-26T02:34:17Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/16603","html_url":"https://github.com/apache/spark/pull/16603","diff_url":"https://github.com/apache/spark/pull/16603.diff","patch_url":"https://github.com/apache/spark/pull/16603.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nIn `TaskMemoryManager `, when we acquire memory by calling `acquireExecutionMemory` and we can't acquire required memory, we will try to spill other memory consumers.\r\n\r\nCurrently, we simply iterates the memory consumers in a hash set. Normally each time the consumer will be iterated in the same order.\r\n\r\nThe first issue is that we might spill additional consumers. For example, if consumer 1 uses 10MB, consumer 2 uses 50MB, then consumer 3 acquires 100MB but we can only get 60MB and spilling is needed. We might spill both consumer 1 and consumer 2. But we actually just need to spill consumer 2 and get the required 100MB.\r\n\r\nThe second issue is that if we spill consumer 1 in first time spilling. After a while, consumer 1 now uses 5MB. Then consumer 4 may acquire some memory and spilling is needed again. Because we iterate the memory consumers in the same order, we will spill consumer 1 again. So for consumer 1, we will produce many small spilling files.\r\n\r\nThis patch modifies the way iterating the memory consumers. It sorts the memory consumers by their memory usage. So the consumer using more memory will spill first. Once it is spilled, even it acquires few memory again, in next time spilling happens it will not be the consumers to spill again if there are other consumers using more memory than it.\r\n\r\n## How was this patch tested?\r\n\r\nJenkins tests.\r\n\r\nPlease review http://spark.apache.org/contributing.html before opening a pull request.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/275294248","html_url":"https://github.com/apache/spark/pull/16603#issuecomment-275294248","issue_url":"https://api.github.com/repos/apache/spark/issues/16603","id":275294248,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars.githubusercontent.com/u/8108735?v=3","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2017-01-26T02:34:17Z","updated_at":"2017-01-26T02:34:17Z","body":"**[Test build #72006 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/72006/testReport)** for PR 16603 at commit [`9f7c8ca`](https://github.com/apache/spark/commit/9f7c8ca8948b07f16327382975276078ea620cf8).\n * This patch **fails Spark unit tests**.\n * This patch merges cleanly.\n * This patch adds no public classes."}}
{"actor":{"display_login":"SparkQA"}}