{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/18730","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/18730/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/18730/comments","events_url":"https://api.github.com/repos/apache/spark/issues/18730/events","html_url":"https://github.com/apache/spark/pull/18730","id":245350657,"number":18730,"title":"[SPARK-21527][CORE] Use buffer limit in order to use JAVA NIO Util's buffercache","user":{"login":"caneGuy","id":26762018,"avatar_url":"https://avatars2.githubusercontent.com/u/26762018?v=4","gravatar_id":"","url":"https://api.github.com/users/caneGuy","html_url":"https://github.com/caneGuy","followers_url":"https://api.github.com/users/caneGuy/followers","following_url":"https://api.github.com/users/caneGuy/following{/other_user}","gists_url":"https://api.github.com/users/caneGuy/gists{/gist_id}","starred_url":"https://api.github.com/users/caneGuy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caneGuy/subscriptions","organizations_url":"https://api.github.com/users/caneGuy/orgs","repos_url":"https://api.github.com/users/caneGuy/repos","events_url":"https://api.github.com/users/caneGuy/events{/privacy}","received_events_url":"https://api.github.com/users/caneGuy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":30,"created_at":"2017-07-25T10:06:07Z","updated_at":"2017-08-25T02:07:23Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/18730","html_url":"https://github.com/apache/spark/pull/18730","diff_url":"https://github.com/apache/spark/pull/18730.diff","patch_url":"https://github.com/apache/spark/pull/18730.patch"},"body":"## What changes were proposed in this pull request?\r\n\r\nRight now, ChunkedByteBuffer#writeFully do not slice bytes first.We observe code in java nio Util#getTemporaryDirectBuffer below:\r\n\r\n\r\n        BufferCache cache = bufferCache.get();\r\n        ByteBuffer buf = cache.get(size);\r\n        if (buf != null) {\r\n            return buf;\r\n        } else {\r\n            // No suitable buffer in the cache so we need to allocate a new\r\n            // one. To avoid the cache growing then we remove the first\r\n            // buffer from the cache and free it.\r\n            if (!cache.isEmpty()) {\r\n                buf = cache.removeFirst();\r\n                free(buf);\r\n            }\r\n            return ByteBuffer.allocateDirect(size);\r\n        }\r\n\r\n\r\nIf we slice first with a fixed size, we can use buffer cache and only need to allocate at the first write call.\r\nSince we allocate new buffer, we can not control the free time of this buffer.This once cause memory issue in our production cluster.\r\nIn this patch, i supply a new api which will slice with fixed size for buffer writing.\r\n\r\n## How was this patch tested?\r\n\r\nUnit test and test in production.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/324804806","html_url":"https://github.com/apache/spark/pull/18730#issuecomment-324804806","issue_url":"https://api.github.com/repos/apache/spark/issues/18730","id":324804806,"user":{"login":"SparkQA","id":8108735,"avatar_url":"https://avatars1.githubusercontent.com/u/8108735?v=4","gravatar_id":"","url":"https://api.github.com/users/SparkQA","html_url":"https://github.com/SparkQA","followers_url":"https://api.github.com/users/SparkQA/followers","following_url":"https://api.github.com/users/SparkQA/following{/other_user}","gists_url":"https://api.github.com/users/SparkQA/gists{/gist_id}","starred_url":"https://api.github.com/users/SparkQA/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SparkQA/subscriptions","organizations_url":"https://api.github.com/users/SparkQA/orgs","repos_url":"https://api.github.com/users/SparkQA/repos","events_url":"https://api.github.com/users/SparkQA/events{/privacy}","received_events_url":"https://api.github.com/users/SparkQA/received_events","type":"User","site_admin":false},"created_at":"2017-08-25T02:07:23Z","updated_at":"2017-08-25T02:07:23Z","body":"**[Test build #81112 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/81112/testReport)** for PR 18730 at commit [`14ca824`](https://github.com/apache/spark/commit/14ca824794ffd543aa169327e78de95f23b1102d)."}}
{"actor":{"display_login":"SparkQA"}}