{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/26269","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/26269/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/26269/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/26269/events","html_url":"https://github.com/elastic/elasticsearch/issues/26269","id":251125976,"number":26269,"title":"How to regulate off-heap usage?","user":{"login":"erik-stephens","id":3054461,"avatar_url":"https://avatars1.githubusercontent.com/u/3054461?v=4","gravatar_id":"","url":"https://api.github.com/users/erik-stephens","html_url":"https://github.com/erik-stephens","followers_url":"https://api.github.com/users/erik-stephens/followers","following_url":"https://api.github.com/users/erik-stephens/following{/other_user}","gists_url":"https://api.github.com/users/erik-stephens/gists{/gist_id}","starred_url":"https://api.github.com/users/erik-stephens/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erik-stephens/subscriptions","organizations_url":"https://api.github.com/users/erik-stephens/orgs","repos_url":"https://api.github.com/users/erik-stephens/repos","events_url":"https://api.github.com/users/erik-stephens/events{/privacy}","received_events_url":"https://api.github.com/users/erik-stephens/received_events","type":"User","site_admin":false},"labels":[{"id":146854632,"url":"https://api.github.com/repos/elastic/elasticsearch/labels/:Network","name":":Network","color":"009800","default":false},{"id":111624690,"url":"https://api.github.com/repos/elastic/elasticsearch/labels/feedback_needed","name":"feedback_needed","color":"d4c5f9","default":false},{"id":582501014,"url":"https://api.github.com/repos/elastic/elasticsearch/labels/v5.5.0","name":"v5.5.0","color":"dddddd","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":37,"created_at":"2017-08-18T02:54:47Z","updated_at":"2017-12-28T16:40:46Z","closed_at":"2017-08-27T19:24:06Z","author_association":"NONE","body":"<!--\r\n\r\n** Please read the guidelines below. **\r\n\r\nIssues that do not follow these guidelines are likely to be closed.\r\n\r\n1.  GitHub is reserved for bug reports and feature requests. The best place to\r\n    ask a general question is at the Elastic [forums](https://discuss.elastic.co).\r\n    GitHub is not the place for general questions.\r\n\r\n2.  Is this bug report or feature request for a supported OS? If not, it\r\n    is likely to be closed.  See https://www.elastic.co/support/matrix#show_os\r\n\r\n3.  Please fill out EITHER the feature request block or the bug report block\r\n    below, and delete the other block.\r\n\r\n-->\r\n\r\n<!-- Bug report -->\r\n\r\n**Elasticsearch version** (`bin/elasticsearch --version`):\r\nVersion: 5.5.0, Build: 260387d/2017-06-30T23:16:05.735Z, JVM: 1.8.0_131\r\n\r\n**Plugins installed**: []\r\n\r\n**JVM version** (`java -version`):\r\nopenjdk version \"1.8.0_66-internal\"\r\nOpenJDK Runtime Environment (build 1.8.0_66-internal-b17)\r\nOpenJDK 64-Bit Server VM (build 25.66-b17, mixed mode)\r\n\r\n**OS version** (`uname -a` if on a Unix-like system):\r\nLinux 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017 x86_64 GNU/Linux\r\n\r\n**Docker version**:\r\nDocker version 1.6.2, build 7c8fca2\r\n\r\n**Description of the problem including expected versus actual behavior**:\r\n\r\nI expect the process RSS to not greatly exceed the sum of -Xmx & -XX:MaxDirectMemorySize.  On a system with 128G of system memory, the elasticsearch process quickly reaches that sum (30G + 32G) and increases slowly until a host level OOM (OS, not JVM).  I've ruled out MetaSpace and things related to threads (num+stack size, thread memory pools).  Please refer to [this thread](https://discuss.elastic.co/t/understanding-off-heap-usage/97176/6) for more background.\r\n\r\n**Steps to reproduce**:\r\n\r\nPlease include a *minimal* but *complete* recreation of the problem, including\r\n(e.g.) index creation, mappings, settings, query etc.  The easier you make for\r\nus to reproduce it, the more likely that somebody will take the time to look at it.\r\n\r\n 1. Start a data-only node with heap & native/direct memory size defined.\r\n 2. The workload is index heavy.  What querying there is handled by tribe nodes.  No aggregations.  This is not a heap issue.\r\n 3. Wait for OOM.\r\n\r\n**Provide logs (if relevant)**:\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/354317377","html_url":"https://github.com/elastic/elasticsearch/issues/26269#issuecomment-354317377","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/26269","id":354317377,"user":{"login":"erik-stephens","id":3054461,"avatar_url":"https://avatars1.githubusercontent.com/u/3054461?v=4","gravatar_id":"","url":"https://api.github.com/users/erik-stephens","html_url":"https://github.com/erik-stephens","followers_url":"https://api.github.com/users/erik-stephens/followers","following_url":"https://api.github.com/users/erik-stephens/following{/other_user}","gists_url":"https://api.github.com/users/erik-stephens/gists{/gist_id}","starred_url":"https://api.github.com/users/erik-stephens/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/erik-stephens/subscriptions","organizations_url":"https://api.github.com/users/erik-stephens/orgs","repos_url":"https://api.github.com/users/erik-stephens/repos","events_url":"https://api.github.com/users/erik-stephens/events{/privacy}","received_events_url":"https://api.github.com/users/erik-stephens/received_events","type":"User","site_admin":false},"created_at":"2017-12-28T16:40:45Z","updated_at":"2017-12-28T16:40:45Z","author_association":"NONE","body":"@xgwu For background, our elasticsearch process is co-hosted with another process that could attempt to allocate a lot of memory quickly.  If your elasticsearch process is the only major process, then I doubt you'll have much risk of an oom situation.\r\n\r\nMy understanding is that this usage is a good thing in that the system is putting unused memory to good use.  It becomes not so good once that usage starts putting pressure on the kernel to allocate new pages quick enough.  If it can't, then it will oom-kill processes, looking at your large elasticsearch process as a good target.  I experimented with increasing `vm.min_free_kbytes` to help leave enough head room for new memory allocations.  Also considered `OOMScoreAdjust` in the elasticsearch systemd unit file as a way to help it survive any oom-kill targeting.\r\n\r\nIn short, it's a linux memory mgmt issue and not an elasticsearch issue.  Some references that helped me:\r\n\r\n- https://unix.stackexchange.com/questions/88693/why-is-swappiness-set-to-60-by-default/88820#88820\r\n- https://jvns.ca/blog/2017/02/17/mystery-swap/\r\n- https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git/tree/mm/vmscan.c#n2152"}}
{"actor":{"display_login":"erik-stephens"}}