{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/18940","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/18940/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/18940/comments","events_url":"https://api.github.com/repos/apache/spark/issues/18940/events","html_url":"https://github.com/apache/spark/pull/18940","id":250064422,"number":18940,"title":"[SPARK-21501] Change CacheLoader to limit entries based on memory footprint","user":{"login":"redsanket","id":8295799,"avatar_url":"https://avatars3.githubusercontent.com/u/8295799?v=4","gravatar_id":"","url":"https://api.github.com/users/redsanket","html_url":"https://github.com/redsanket","followers_url":"https://api.github.com/users/redsanket/followers","following_url":"https://api.github.com/users/redsanket/following{/other_user}","gists_url":"https://api.github.com/users/redsanket/gists{/gist_id}","starred_url":"https://api.github.com/users/redsanket/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/redsanket/subscriptions","organizations_url":"https://api.github.com/users/redsanket/orgs","repos_url":"https://api.github.com/users/redsanket/repos","events_url":"https://api.github.com/users/redsanket/events{/privacy}","received_events_url":"https://api.github.com/users/redsanket/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":15,"created_at":"2017-08-14T15:21:37Z","updated_at":"2017-08-15T22:01:26Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/18940","html_url":"https://github.com/apache/spark/pull/18940","diff_url":"https://github.com/apache/spark/pull/18940.diff","patch_url":"https://github.com/apache/spark/pull/18940.patch"},"body":"Right now the spark shuffle service has a cache for index files. It is based on a # of files cached (spark.shuffle.service.index.cache.entries). This can cause issues if people have a lot of reducers because the size of each entry can fluctuate based on the # of reducers.\r\nWe saw an issues with a job that had 170000 reducers and it caused NM with spark shuffle service to use 700-800MB or memory in NM by itself.\r\nWe should change this cache to be memory based and only allow a certain memory size used. When I say memory based I mean the cache should have a limit of say 100MB.\r\n\r\nhttps://issues.apache.org/jira/browse/SPARK-21501\r\n\r\nManual Testing with 170000 reducers has been performed with cache loaded up to max 100MB default limit, with each shuffle index file of size 1.3MB. Eviction takes place as soon as the total cache size reaches the 100MB limit and the objects will be ready for garbage collection there by avoiding NM to crash. No notable difference in runtime has been observed."},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/322602066","html_url":"https://github.com/apache/spark/pull/18940#issuecomment-322602066","issue_url":"https://api.github.com/repos/apache/spark/issues/18940","id":322602066,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars3.githubusercontent.com/u/4031291?v=4","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2017-08-15T22:01:26Z","updated_at":"2017-08-15T22:01:26Z","body":"Test PASSed.\nRefer to this link for build results (access rights to CI server needed): \nhttps://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80696/\nTest PASSed."}}
{"actor":{"display_login":"AmplabJenkins"}}