{"action":"created","issue":{"url":"https://api.github.com/repos/dask/dask/issues/2551","repository_url":"https://api.github.com/repos/dask/dask","labels_url":"https://api.github.com/repos/dask/dask/issues/2551/labels{/name}","comments_url":"https://api.github.com/repos/dask/dask/issues/2551/comments","events_url":"https://api.github.com/repos/dask/dask/issues/2551/events","html_url":"https://github.com/dask/dask/issues/2551","id":245411952,"number":2551,"title":"ResourceProfile has different behaviour using context manager vs `register`","user":{"login":"bmerry","id":1963944,"avatar_url":"https://avatars1.githubusercontent.com/u/1963944?v=4","gravatar_id":"","url":"https://api.github.com/users/bmerry","html_url":"https://github.com/bmerry","followers_url":"https://api.github.com/users/bmerry/followers","following_url":"https://api.github.com/users/bmerry/following{/other_user}","gists_url":"https://api.github.com/users/bmerry/gists{/gist_id}","starred_url":"https://api.github.com/users/bmerry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bmerry/subscriptions","organizations_url":"https://api.github.com/users/bmerry/orgs","repos_url":"https://api.github.com/users/bmerry/repos","events_url":"https://api.github.com/users/bmerry/events{/privacy}","received_events_url":"https://api.github.com/users/bmerry/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-07-25T14:02:20Z","updated_at":"2017-07-25T14:58:11Z","closed_at":null,"body":"I'm using the dask profilers with the pattern\r\n```\r\nfor profiler in profilers: profiler.register()\r\nfoo() # Do work, some of it dask, some of it other\r\nfor profiler in profilers: profiler.unregister()\r\n```\r\n(although with some try/finally stuff to make it more robust). I had assumed that this would be equivalent to running `foo` under a context manager per profiler, except that the `with` statement doesn't allow for unpacking so can't take a variable number of context managers (and `contextlib.nested` is deprecated and I'm on Python 2 so can't use `ExitStack`). However, the `ResourceProfiler` code has specific checks for whether it is currently being used as a context manager (`_enter` member) and if not, it stops tracking resources as soon as a computation is complete (in `_finish`), which leads to gaps in the graph whenever no dask computation is running.\r\n\r\nExample code:\r\n```python\r\n#!/usr/bin/env python\r\nimport dask.array as da\r\nfrom dask.diagnostics import ResourceProfiler\r\nimport time\r\n\r\ndef foo():\r\n    x = da.ones(1000000000, chunks=1000000)\r\n    da.dot(x, x).compute()  # Keep things busy for a while\r\n    time.sleep(3)\r\n    da.dot(x, x).compute()  # Keep things busy again\r\n\r\nprofiler = ResourceProfiler(dt=0.2)\r\nprofiler.register()\r\nfoo()\r\nprofiler.unregister()\r\nprofiler.visualize()\r\n```"},"comment":{"url":"https://api.github.com/repos/dask/dask/issues/comments/317765039","html_url":"https://github.com/dask/dask/issues/2551#issuecomment-317765039","issue_url":"https://api.github.com/repos/dask/dask/issues/2551","id":317765039,"user":{"login":"jcrist","id":2783717,"avatar_url":"https://avatars3.githubusercontent.com/u/2783717?v=4","gravatar_id":"","url":"https://api.github.com/users/jcrist","html_url":"https://github.com/jcrist","followers_url":"https://api.github.com/users/jcrist/followers","following_url":"https://api.github.com/users/jcrist/following{/other_user}","gists_url":"https://api.github.com/users/jcrist/gists{/gist_id}","starred_url":"https://api.github.com/users/jcrist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jcrist/subscriptions","organizations_url":"https://api.github.com/users/jcrist/orgs","repos_url":"https://api.github.com/users/jcrist/repos","events_url":"https://api.github.com/users/jcrist/events{/privacy}","received_events_url":"https://api.github.com/users/jcrist/received_events","type":"User","site_admin":false},"created_at":"2017-07-25T14:58:11Z","updated_at":"2017-07-25T14:58:11Z","body":"> except that the with statement doesn't allow for unpacking so can't take a variable number of context managers\r\n\r\nFor non variable numbers, you can do:\r\n\r\n```\r\nwith ResourceProfiler() as rprof, Profiler( )as prof:\r\n    ...\r\n```\r\n\r\nwhich is my common pattern.\r\n\r\n> it stops tracking resources as soon as a computation is complete (in _finish)\r\n\r\nThis mirrors the other profilers, which also only track during dask computation. As a context manager the `ResourceProfiler` is actually an exception, as it can be used to profile non-dask code (always tracks around a context). I'm slightly against changing this behavior. What would you want this to be instead?"}}
{"actor":{"display_login":"jcrist"}}