{"action":"created","issue":{"url":"https://api.github.com/repos/jemalloc/jemalloc/issues/892","repository_url":"https://api.github.com/repos/jemalloc/jemalloc","labels_url":"https://api.github.com/repos/jemalloc/jemalloc/issues/892/labels{/name}","comments_url":"https://api.github.com/repos/jemalloc/jemalloc/issues/892/comments","events_url":"https://api.github.com/repos/jemalloc/jemalloc/issues/892/events","html_url":"https://github.com/jemalloc/jemalloc/issues/892","id":234249798,"number":892,"title":"Question on jemalloc stats.resident and reason of physical memory growth beyond that","user":{"login":"yliu198","id":29256320,"avatar_url":"https://avatars2.githubusercontent.com/u/29256320?v=3","gravatar_id":"","url":"https://api.github.com/users/yliu198","html_url":"https://github.com/yliu198","followers_url":"https://api.github.com/users/yliu198/followers","following_url":"https://api.github.com/users/yliu198/following{/other_user}","gists_url":"https://api.github.com/users/yliu198/gists{/gist_id}","starred_url":"https://api.github.com/users/yliu198/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yliu198/subscriptions","organizations_url":"https://api.github.com/users/yliu198/orgs","repos_url":"https://api.github.com/users/yliu198/repos","events_url":"https://api.github.com/users/yliu198/events{/privacy}","received_events_url":"https://api.github.com/users/yliu198/received_events","type":"User","site_admin":false},"labels":[{"id":59304344,"url":"https://api.github.com/repos/jemalloc/jemalloc/labels/question","name":"question","color":"cc317c","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-06-07T15:28:31Z","updated_at":"2017-06-07T17:31:42Z","closed_at":null,"body":"Hi, \r\n\r\nI am trying to use jemalloc and its stats/profiling capability to investigate my application's memory growth issue. \r\n\r\nIn my testing, jemalloc usually works well in terms of keeping physical memory footprint low and releasing deallocated memory back to the OS (i.e. freeing physical memory). (So thanks for the great work!)\r\n\r\nHowever, sometimes my application's physical memory started to linearly grow for unknown reason by a few Giga-bytes in an hour or two, which is puzzling me. \r\n\r\nDuring such a period, I collected jemalloc stats as follows:\r\n---Before the growth:\r\n05/18 23:04:51\r\nAllocated:\t459652176\tactive:\t568348672\tmetadata:\t120945664\tresident:\t814112768\tmapped:\t4487905280\tretained:\t3695181824\t\t\t8183087104\r\n---After the growth:\r\n05/19 02:04:52\r\nAllocated:\t459689248\tactive:\t568401920\tmetadata:\t120945664\tresident:\t814153728\tmapped:\t4487905280\tretained:\t3695181824\t\t\t8183087104\r\n---\r\nBasically, \"Allocated\" increased 37072, \"active\" increased 53248, \"resident\" increased 40960 and rest unchanged. However, meanwhile the OS reported resident memory of the process increased from ~1GBytes to ~4.4GBytes. Btw, I enabled jemalloc profiler in the test and it did not report large amount of malloc activities during the period.\r\n\r\nHere are my questions: \r\n(1) Can jemalloc's stats underestimate the \"resident\" memory amount used by an application at all, if so can it be so much? \r\n(2) Otherwise, what could be the possible causes for my applications' strange physical memory growth? I suspect it is some bug that accessed free'ed memory during the period, thus caused the OS to re-map physical pages for large amount of free virtual memory, but unfortunately have not found such bug yet. Is this a possible cause? \r\n(3) Unlikely, but could there be some logic in my application that caused memory fragmentation?\r\n\r\nMy application runs on x86_64 linux with kernel version 2.6.32. \r\n\r\nAny suggestion will be appreciated!\r\n\r\n-Yong"},"comment":{"url":"https://api.github.com/repos/jemalloc/jemalloc/issues/comments/306867865","html_url":"https://github.com/jemalloc/jemalloc/issues/892#issuecomment-306867865","issue_url":"https://api.github.com/repos/jemalloc/jemalloc/issues/892","id":306867865,"user":{"login":"jasone","id":245970,"avatar_url":"https://avatars2.githubusercontent.com/u/245970?v=3","gravatar_id":"","url":"https://api.github.com/users/jasone","html_url":"https://github.com/jasone","followers_url":"https://api.github.com/users/jasone/followers","following_url":"https://api.github.com/users/jasone/following{/other_user}","gists_url":"https://api.github.com/users/jasone/gists{/gist_id}","starred_url":"https://api.github.com/users/jasone/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jasone/subscriptions","organizations_url":"https://api.github.com/users/jasone/orgs","repos_url":"https://api.github.com/users/jasone/repos","events_url":"https://api.github.com/users/jasone/events{/privacy}","received_events_url":"https://api.github.com/users/jasone/received_events","type":"User","site_admin":false},"created_at":"2017-06-07T17:31:42Z","updated_at":"2017-06-07T17:31:42Z","body":"Are there any mapped files that could explain the growth?  If not, the only other idea I have is that transparent huge pages are causing physical memory usage to be much higher than jemalloc's estimate, which is based on 4 KiB pages rather than 2 MiB pages."}}
{"actor":{"display_login":"jasone"}}