{"action":"created","issue":{"url":"https://api.github.com/repos/GoogleCloudPlatform/python-compat-runtime/issues/95","repository_url":"https://api.github.com/repos/GoogleCloudPlatform/python-compat-runtime","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/python-compat-runtime/issues/95/labels{/name}","comments_url":"https://api.github.com/repos/GoogleCloudPlatform/python-compat-runtime/issues/95/comments","events_url":"https://api.github.com/repos/GoogleCloudPlatform/python-compat-runtime/issues/95/events","html_url":"https://github.com/GoogleCloudPlatform/python-compat-runtime/issues/95","id":156028742,"number":95,"title":"multiprocessing.Pool workers hold on to response body","user":{"login":"evanj","id":675151,"avatar_url":"https://avatars1.githubusercontent.com/u/675151?v=3","gravatar_id":"","url":"https://api.github.com/users/evanj","html_url":"https://github.com/evanj","followers_url":"https://api.github.com/users/evanj/followers","following_url":"https://api.github.com/users/evanj/following{/other_user}","gists_url":"https://api.github.com/users/evanj/gists{/gist_id}","starred_url":"https://api.github.com/users/evanj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evanj/subscriptions","organizations_url":"https://api.github.com/users/evanj/orgs","repos_url":"https://api.github.com/users/evanj/repos","events_url":"https://api.github.com/users/evanj/events{/privacy}","received_events_url":"https://api.github.com/users/evanj/received_events","type":"User","site_admin":false},"labels":[{"id":171453318,"url":"https://api.github.com/repos/GoogleCloudPlatform/python-compat-runtime/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":{"login":"duggelz","id":608338,"avatar_url":"https://avatars0.githubusercontent.com/u/608338?v=3","gravatar_id":"","url":"https://api.github.com/users/duggelz","html_url":"https://github.com/duggelz","followers_url":"https://api.github.com/users/duggelz/followers","following_url":"https://api.github.com/users/duggelz/following{/other_user}","gists_url":"https://api.github.com/users/duggelz/gists{/gist_id}","starred_url":"https://api.github.com/users/duggelz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/duggelz/subscriptions","organizations_url":"https://api.github.com/users/duggelz/orgs","repos_url":"https://api.github.com/users/duggelz/repos","events_url":"https://api.github.com/users/duggelz/events{/privacy}","received_events_url":"https://api.github.com/users/duggelz/received_events","type":"User","site_admin":false},"assignees":[{"login":"duggelz","id":608338,"avatar_url":"https://avatars0.githubusercontent.com/u/608338?v=3","gravatar_id":"","url":"https://api.github.com/users/duggelz","html_url":"https://github.com/duggelz","followers_url":"https://api.github.com/users/duggelz/followers","following_url":"https://api.github.com/users/duggelz/following{/other_user}","gists_url":"https://api.github.com/users/duggelz/gists{/gist_id}","starred_url":"https://api.github.com/users/duggelz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/duggelz/subscriptions","organizations_url":"https://api.github.com/users/duggelz/orgs","repos_url":"https://api.github.com/users/duggelz/repos","events_url":"https://api.github.com/users/duggelz/events{/privacy}","received_events_url":"https://api.github.com/users/duggelz/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/GoogleCloudPlatform/python-compat-runtime/milestones/1","html_url":"https://github.com/GoogleCloudPlatform/python-compat-runtime/milestone/1","labels_url":"https://api.github.com/repos/GoogleCloudPlatform/python-compat-runtime/milestones/1/labels","id":1725842,"number":1,"title":"Beta","description":"","creator":{"login":"jonparrott","id":250995,"avatar_url":"https://avatars1.githubusercontent.com/u/250995?v=3","gravatar_id":"","url":"https://api.github.com/users/jonparrott","html_url":"https://github.com/jonparrott","followers_url":"https://api.github.com/users/jonparrott/followers","following_url":"https://api.github.com/users/jonparrott/following{/other_user}","gists_url":"https://api.github.com/users/jonparrott/gists{/gist_id}","starred_url":"https://api.github.com/users/jonparrott/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jonparrott/subscriptions","organizations_url":"https://api.github.com/users/jonparrott/orgs","repos_url":"https://api.github.com/users/jonparrott/repos","events_url":"https://api.github.com/users/jonparrott/events{/privacy}","received_events_url":"https://api.github.com/users/jonparrott/received_events","type":"User","site_admin":false},"open_issues":1,"closed_issues":7,"state":"open","created_at":"2016-04-22T19:26:31Z","updated_at":"2016-05-20T19:02:25Z","due_on":null,"closed_at":null},"comments":5,"created_at":"2016-05-20T18:55:59Z","updated_at":"2017-05-31T02:17:10Z","closed_at":null,"body":"In our app, we make use of pull task queues, and we pull large number of tasks with large bodies (e.g. ~5 MB of payloads). I noticed that it appeared we had a memory leak when running this in the `python-compat` runtime, with many megabytes of strings being retained. With a whole lot of hacking, I ended up tracking it down to the following:\n- Multiprocessing pool workers hold on to the result objects in their local variables until they get another work item.\n- The pool has 100 threads in the python-compat environment. This means it can retain up to ~100 items.\n- The requests.Response is the result type that gets cached, and it stores the body of the response.\n\nMy hack to fix it: in `google/appengine/ext/vmruntime/vmstub.py`, I set `response._content = None` right after the response protocol buffer message is parsed. This caused our process to use ~100 MB of memory, instead of ~400 MB of memory.\n\nHacky output from my tool that found the large strings that were being retained, showing what is holding on to it:\n\n```\nfound new string! len=4653338\nreferred by 139691536297608 <type 'dict'>\n-referred by 139691536340816 <class 'requests_nologs.models.Response'>\n--referred by 139691536378120 <type 'tuple'>\n---referred by frame /usr/lib/python2.7/multiprocessing/pool.py 102 worker\n---referred by frame /usr/lib/python2.7/multiprocessing/pool.py 380 _handle_results\n```\n"},"comment":{"url":"https://api.github.com/repos/GoogleCloudPlatform/python-compat-runtime/issues/comments/305062897","html_url":"https://github.com/GoogleCloudPlatform/python-compat-runtime/issues/95#issuecomment-305062897","issue_url":"https://api.github.com/repos/GoogleCloudPlatform/python-compat-runtime/issues/95","id":305062897,"user":{"login":"mikelambert","id":7953259,"avatar_url":"https://avatars2.githubusercontent.com/u/7953259?v=3","gravatar_id":"","url":"https://api.github.com/users/mikelambert","html_url":"https://github.com/mikelambert","followers_url":"https://api.github.com/users/mikelambert/followers","following_url":"https://api.github.com/users/mikelambert/following{/other_user}","gists_url":"https://api.github.com/users/mikelambert/gists{/gist_id}","starred_url":"https://api.github.com/users/mikelambert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikelambert/subscriptions","organizations_url":"https://api.github.com/users/mikelambert/orgs","repos_url":"https://api.github.com/users/mikelambert/repos","events_url":"https://api.github.com/users/mikelambert/events{/privacy}","received_events_url":"https://api.github.com/users/mikelambert/received_events","type":"User","site_admin":false},"created_at":"2017-05-31T02:17:10Z","updated_at":"2017-05-31T02:17:10Z","body":"FWIW this bug still manifests even outside `python-compat` and `vmstub.py`, just to a lesser extent (since it's `100` instead of `100*100`).\r\n\r\nBut just to follow-up: adding in the fixed `multiprocessing/pool.py` appears to have significantly helped my memory issues.\r\n\r\nPreviously I would inch up to 100% memory usage within 24 hours (+0.3GB/hour), and then eventually swap-slowly, crash, and restart. Now it seems to be consistently hovering around 27%.\r\n"}}
{"actor":{"display_login":"mikelambert"}}