{"action":"created","issue":{"url":"https://api.github.com/repos/druid-io/druid/issues/4422","repository_url":"https://api.github.com/repos/druid-io/druid","labels_url":"https://api.github.com/repos/druid-io/druid/issues/4422/labels{/name}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/4422/comments","events_url":"https://api.github.com/repos/druid-io/druid/issues/4422/events","html_url":"https://github.com/druid-io/druid/issues/4422","id":236636850,"number":4422,"title":"Agile, config-less approach to off-heap memory management","user":{"login":"leventov","id":609240,"avatar_url":"https://avatars0.githubusercontent.com/u/609240?v=4","gravatar_id":"","url":"https://api.github.com/users/leventov","html_url":"https://github.com/leventov","followers_url":"https://api.github.com/users/leventov/followers","following_url":"https://api.github.com/users/leventov/following{/other_user}","gists_url":"https://api.github.com/users/leventov/gists{/gist_id}","starred_url":"https://api.github.com/users/leventov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leventov/subscriptions","organizations_url":"https://api.github.com/users/leventov/orgs","repos_url":"https://api.github.com/users/leventov/repos","events_url":"https://api.github.com/users/leventov/events{/privacy}","received_events_url":"https://api.github.com/users/leventov/received_events","type":"User","site_admin":false},"labels":[{"id":219585748,"url":"https://api.github.com/repos/druid-io/druid/labels/Area%20-%20Querying","name":"Area - Querying","color":"fef2c0","default":false},{"id":120288359,"url":"https://api.github.com/repos/druid-io/druid/labels/Performance","name":"Performance","color":"fbca04","default":false},{"id":628450570,"url":"https://api.github.com/repos/druid-io/druid/labels/Proposal","name":"Proposal","color":"e99695","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2017-06-17T03:56:19Z","updated_at":"2017-09-29T20:21:35Z","closed_at":null,"author_association":"MEMBER","body":"Currently we have a pool of large (usually they are configured so) buffers, on each query that needs off-heap memory one buffer is allocated for the query. If the query requires more off-heap memory that the buffer size, the data is processed in several runs, each run on a on a subset of dimension values, clearing the buffer between the runs.\r\n\r\nProblems:\r\n - lack of agility, if the buffer size is configured too small, queries which require a lot off-heap memory are processed in many runs, i. e. too slow, while there could actually be enough free off-heap memory in the system at the moment to process the query in a little number of runs, or in a single run. On the other hand, buffer size couldn't be too big, the limit is the total amount of off-heap memory available (total memory dedicated for the Druid instance - heap memory size) divided by the number of processing threads.\r\n - need to configure bufferSize and number of buffers in the first place. Configs are evil. Sometimes users configure inadequate values and don't understand why the system perform poorly.\r\n\r\n### The different approach\r\nOff-heap memory is allocated in relatively small chunks, fixed sized, e. g. 64K. Fixed chunk size is needed to avoid fragmentation, and to allow simple, lock-free allocation logic.\r\n\r\nWhen a query needs more off-heap memory, it is provided with a _list of allocated chunks_. But it is not a list of separate `ByteBuffer` objects (it would kill query processing performance), it is a list of offsets within a single giant ByteBuffer.\r\n\r\nCurrently space allocation for a dimension value [looks like this](https://github.com/druid-io/druid/blob/master/processing/src/main/java/io/druid/query/topn/PooledTopNAlgorithm.java#L669-L670):\r\n```java\r\npositions[dimIndex] = currentPosition * numBytesPerRecord;\r\ncurrentPosition++;\r\n```\r\n\r\nWill be just a little more complex:\r\n```java\r\nif (currentChunkOffset + numBytesPerRecord > currentChunkLimit) {\r\n  currentChunkOffset = listOfChunks.get(chunkIndex++);\r\n  currentChunkLimit = currentChunkOffset + chunkSize;\r\n}\r\npositions[dimIndex] = currentChunkOffset;\r\ncurrentChunkOffset += numBytesPerRecord;\r\n```\r\n\r\nOtherwise the query processing speed shouldn't be affected.\r\n\r\nWhat problems it solves:\r\n - Only a single config needed, that is much easier to get right: the total amount of memory, that user dedicates for off-heap processing on the Druid instance.\r\n - On heterogeneous query loads, resources are used more effectively.\r\n\r\n#### Problem with this approach\r\n\r\nCurrently ByteBuffer size is limited to 2 GB, that complicates resource sharing. It would be better to allocate a single giant memory area for all off-heap memory. #3892 will solve this."},"comment":{"url":"https://api.github.com/repos/druid-io/druid/issues/comments/333229403","html_url":"https://github.com/druid-io/druid/issues/4422#issuecomment-333229403","issue_url":"https://api.github.com/repos/druid-io/druid/issues/4422","id":333229403,"user":{"login":"leventov","id":609240,"avatar_url":"https://avatars0.githubusercontent.com/u/609240?v=4","gravatar_id":"","url":"https://api.github.com/users/leventov","html_url":"https://github.com/leventov","followers_url":"https://api.github.com/users/leventov/followers","following_url":"https://api.github.com/users/leventov/following{/other_user}","gists_url":"https://api.github.com/users/leventov/gists{/gist_id}","starred_url":"https://api.github.com/users/leventov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leventov/subscriptions","organizations_url":"https://api.github.com/users/leventov/orgs","repos_url":"https://api.github.com/users/leventov/repos","events_url":"https://api.github.com/users/leventov/events{/privacy}","received_events_url":"https://api.github.com/users/leventov/received_events","type":"User","site_admin":false},"created_at":"2017-09-29T20:21:35Z","updated_at":"2017-09-29T20:21:35Z","author_association":"MEMBER","body":"GroupBy and decompression use different sets of buffers already. Also in a long run, decompression buffers should not be needed, see https://github.com/druid-io/druid/issues/4080"}}
{"actor":{"display_login":"leventov"}}