{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/2470","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/2470/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/2470/comments","events_url":"https://api.github.com/repos/dask/dask/issues/2470/events","html_url":"https://github.com/dask/dask/issues/2470","id":236638229,"number":2470,"title":"Running out of memory due to task prioritization by scheduler","user":{"login":"olivier-lacroix","id":24560122,"avatar_url":"https://avatars0.githubusercontent.com/u/24560122?v=4","gravatar_id":"","url":"https://api.github.com/users/olivier-lacroix","html_url":"https://github.com/olivier-lacroix","followers_url":"https://api.github.com/users/olivier-lacroix/followers","following_url":"https://api.github.com/users/olivier-lacroix/following{/other_user}","gists_url":"https://api.github.com/users/olivier-lacroix/gists{/gist_id}","starred_url":"https://api.github.com/users/olivier-lacroix/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olivier-lacroix/subscriptions","organizations_url":"https://api.github.com/users/olivier-lacroix/orgs","repos_url":"https://api.github.com/users/olivier-lacroix/repos","events_url":"https://api.github.com/users/olivier-lacroix/events{/privacy}","received_events_url":"https://api.github.com/users/olivier-lacroix/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2017-06-17T04:32:23Z","updated_at":"2017-10-20T16:55:11Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\n\r\nI am using dask (0.14.3) (Python 3.6.1 :: Anaconda 4.4.0 (64-bit)) and have run into an issue using either the distributed or multiprocessing scheduler on a single machine.\r\n\r\nI am applying a function through a `map_partitions` on a large dask dataframe (read from several h5 files) that returns a result with a small RAM footprint from a relatively large dask partition, and then `compute` the result. \r\n\r\nDoing this, the memory footprint increases until the system runs out of it and the kernel kills a couple of workers. Looking at task progress with the distributed scheduler, a lot of partitions are being read in advance (2-3 per worker), and then wait in RAM for the function to be applied, and thus filling up the RAM unnecessarily.\r\n\r\nIs there a way to avoid this suicidal behaviour from the scheduler(s)?\r\n\r\nNote: the only way around it I found was reducing the number of workers, which is not ideal.\r\n\r\nThanks\r\n\r\nOlivier"},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/338262880","html_url":"https://github.com/dask/dask/issues/2470#issuecomment-338262880","issue_url":"https://api.github.com/repos/dask/dask/issues/2470","id":338262880,"user":{"login":"ogrisel","id":89061,"avatar_url":"https://avatars0.githubusercontent.com/u/89061?v=4","gravatar_id":"","url":"https://api.github.com/users/ogrisel","html_url":"https://github.com/ogrisel","followers_url":"https://api.github.com/users/ogrisel/followers","following_url":"https://api.github.com/users/ogrisel/following{/other_user}","gists_url":"https://api.github.com/users/ogrisel/gists{/gist_id}","starred_url":"https://api.github.com/users/ogrisel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ogrisel/subscriptions","organizations_url":"https://api.github.com/users/ogrisel/orgs","repos_url":"https://api.github.com/users/ogrisel/repos","events_url":"https://api.github.com/users/ogrisel/events{/privacy}","received_events_url":"https://api.github.com/users/ogrisel/received_events","type":"User","site_admin":false},"created_at":"2017-10-20T16:55:11Z","updated_at":"2017-10-20T16:55:11Z","author_association":"CONTRIBUTOR","body":"Aggressively triggering GC when detecting high memory usage in the worker seem to fix the issue when using the 1GB boundary. I will submit a PR."}}
{"actor":{"display_login":"ogrisel"}}