{"action":"created","issue":{"url":"https://api.github.com/repos/dask/distributed/issues/966","repository_url":"https://api.github.com/repos/dask/distributed","labels_url":"https://api.github.com/repos/dask/distributed/issues/966/labels{/name}","comments_url":"https://api.github.com/repos/dask/distributed/issues/966/comments","events_url":"https://api.github.com/repos/dask/distributed/issues/966/events","html_url":"https://github.com/dask/distributed/issues/966","id":216508145,"number":966,"title":"Intermittent testing failures","user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars3.githubusercontent.com/u/306380?v=4","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":25,"created_at":"2017-03-23T17:22:38Z","updated_at":"2017-12-04T18:57:17Z","closed_at":null,"author_association":"OWNER","body":"Some tests fail very infrequently.  Lets maintain a list here:\r\n\r\n- [ ] distributed/cli/tests/test_dask_worker.py::test_local_directory\r\n- [ ] distributed/tests/test_ipython.py::test_start_ipython_scheduler (windows)\r\n- [ ] distributed/deploy/tests/test_local.py::test_silent_startup [appveyor](https://ci.appveyor.com/project/daskdev/distributed/build/1.0.1299#L1535)\r\n"},"comment":{"url":"https://api.github.com/repos/dask/distributed/issues/comments/349067261","html_url":"https://github.com/dask/distributed/issues/966#issuecomment-349067261","issue_url":"https://api.github.com/repos/dask/distributed/issues/966","id":349067261,"user":{"login":"mrocklin","id":306380,"avatar_url":"https://avatars3.githubusercontent.com/u/306380?v=4","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"created_at":"2017-12-04T18:57:17Z","updated_at":"2017-12-04T18:57:17Z","author_association":"OWNER","body":"`test_restart_scheduler`\r\n\r\n<details>\r\n\r\n```python\r\n___________________________ test_restart_scheduler ____________________________\r\nTraceback (most recent call last):\r\n  File \"C:\\projects\\distributed\\distributed\\utils_test.py\", line 723, in test_func\r\n    result = loop.run_sync(coro, timeout=timeout)\r\n  File \"C:\\Miniconda35-x64\\envs\\testenv\\lib\\site-packages\\tornado\\ioloop.py\", line 458, in run_sync\r\n    return future_cell[0].result()\r\n  File \"C:\\Miniconda35-x64\\envs\\testenv\\lib\\site-packages\\tornado\\concurrent.py\", line 238, in result\r\n    raise_exc_info(self._exc_info)\r\n  File \"<string>\", line 4, in raise_exc_info\r\n  File \"C:\\Miniconda35-x64\\envs\\testenv\\lib\\site-packages\\tornado\\gen.py\", line 1069, in run\r\n    yielded = self.gen.send(value)\r\n  File \"C:\\projects\\distributed\\distributed\\utils_test.py\", line 711, in coro\r\n    result = yield func(*args)\r\n  File \"C:\\Miniconda35-x64\\envs\\testenv\\lib\\site-packages\\tornado\\gen.py\", line 1055, in run\r\n    value = future.result()\r\n  File \"C:\\Miniconda35-x64\\envs\\testenv\\lib\\site-packages\\tornado\\concurrent.py\", line 238, in result\r\n    raise_exc_info(self._exc_info)\r\n  File \"<string>\", line 4, in raise_exc_info\r\n  File \"C:\\Miniconda35-x64\\envs\\testenv\\lib\\site-packages\\tornado\\gen.py\", line 1063, in run\r\n    yielded = self.gen.throw(*exc_info)\r\n  File \"C:\\projects\\distributed\\distributed\\tests\\test_worker_failure.py\", line 279, in test_restart_scheduler\r\n    yield s.restart()\r\n  File \"C:\\Miniconda35-x64\\envs\\testenv\\lib\\site-packages\\tornado\\gen.py\", line 1055, in run\r\n    value = future.result()\r\n  File \"C:\\Miniconda35-x64\\envs\\testenv\\lib\\site-packages\\tornado\\concurrent.py\", line 238, in result\r\n    raise_exc_info(self._exc_info)\r\n  File \"<string>\", line 4, in raise_exc_info\r\n  File \"C:\\Miniconda35-x64\\envs\\testenv\\lib\\site-packages\\tornado\\gen.py\", line 1069, in run\r\n    yielded = self.gen.send(value)\r\n  File \"C:\\projects\\distributed\\distributed\\scheduler.py\", line 1678, in restart\r\n    assert all(resp == 'OK' for resp in resps)\r\nAssertionError\r\n---------------------------- Captured stderr call -----------------------------\r\ndistributed.scheduler - INFO -   Scheduler at:     tcp://127.0.0.1:16038\r\ndistributed.nanny - INFO -         Start Nanny at: 'tcp://127.0.0.1:16039'\r\ndistributed.nanny - INFO -         Start Nanny at: 'tcp://127.0.0.1:16040'\r\ndistributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:16045\r\ndistributed.worker - INFO -          Listening to:      tcp://127.0.0.1:16045\r\ndistributed.worker - INFO -              nanny at:            127.0.0.1:16039\r\ndistributed.worker - INFO - Waiting to connect to:      tcp://127.0.0.1:16038\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -               Threads:                          1\r\ndistributed.worker - INFO -                Memory:                    1.82 GB\r\ndistributed.worker - INFO -       Local Directory: C:\\projects\\distributed\\dask-worker-space\\worker-p91_e4b0\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.scheduler - INFO - Register tcp://127.0.0.1:16045\r\ndistributed.worker - INFO -         Registered to:      tcp://127.0.0.1:16038\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:16045\r\ndistributed.core - WARNING - Event loop was unresponsive for 1.03s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.\r\ndistributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:16048\r\ndistributed.worker - INFO -          Listening to:      tcp://127.0.0.1:16048\r\ndistributed.worker - INFO -              nanny at:            127.0.0.1:16040\r\ndistributed.worker - INFO - Waiting to connect to:      tcp://127.0.0.1:16038\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -               Threads:                          2\r\ndistributed.worker - INFO -                Memory:                    1.82 GB\r\ndistributed.worker - INFO -       Local Directory: C:\\projects\\distributed\\dask-worker-space\\worker-xeo4vie7\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.scheduler - INFO - Register tcp://127.0.0.1:16048\r\ndistributed.worker - INFO -         Registered to:      tcp://127.0.0.1:16038\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:16048\r\ndistributed.scheduler - INFO - Send lost future signal to clients\r\ndistributed.scheduler - INFO - Remove worker tcp://127.0.0.1:16045\r\ndistributed.scheduler - INFO - Remove worker tcp://127.0.0.1:16048\r\ndistributed.scheduler - INFO - Lost all workers\r\ndistributed.scheduler - INFO - Clear task state\r\ndistributed.core - WARNING - Event loop was unresponsive for 1.49s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.\r\ndistributed.worker - INFO - Stopping worker at tcp://127.0.0.1:16045\r\ndistributed.worker - INFO - Stopping worker at tcp://127.0.0.1:16048\r\ndistributed.nanny - WARNING - Worker process still alive after 1 seconds, killing\r\ndistributed.nanny - WARNING - Worker process still alive after 1 seconds, killing\r\ndistributed.nanny - ERROR - Restart timed out, returning before finished\r\ndistributed.nanny - ERROR - Restart timed out, returning before finished\r\ndistributed.utils - ERROR - \r\nTraceback (most recent call last):\r\n  File \"C:\\projects\\distributed\\distributed\\utils.py\", line 622, in log_errors\r\n    yield\r\n  File \"C:\\projects\\distributed\\distributed\\scheduler.py\", line 1678, in restart\r\n    assert all(resp == 'OK' for resp in resps)\r\nAssertionError\r\ndistributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:16039'\r\ndistributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:16040'\r\ndistributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:16059\r\ndistributed.worker - INFO -          Listening to:      tcp://127.0.0.1:16059\r\ndistributed.worker - INFO -              nanny at:            127.0.0.1:16039\r\ndistributed.worker - INFO - Waiting to connect to:      tcp://127.0.0.1:16038\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -               Threads:                          1\r\ndistributed.worker - INFO -                Memory:                    1.82 GB\r\ndistributed.worker - INFO -       Local Directory: C:\\projects\\distributed\\dask-worker-space\\worker-wmi397wc\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO - Stopping worker at tcp://127.0.0.1:16059\r\ndistributed.diskutils - WARNING - Found stale lock file and directory 'C:\\\\projects\\\\distributed\\\\dask-worker-space\\\\worker-xeo4vie7', purging\r\ndistributed.worker - INFO -       Start worker at:      tcp://127.0.0.1:16061\r\ndistributed.worker - INFO -          Listening to:      tcp://127.0.0.1:16061\r\ndistributed.worker - INFO -              nanny at:            127.0.0.1:16040\r\ndistributed.worker - INFO - Waiting to connect to:      tcp://127.0.0.1:16038\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO -               Threads:                          2\r\ndistributed.worker - INFO -                Memory:                    1.82 GB\r\ndistributed.worker - INFO -       Local Directory: C:\\projects\\distributed\\dask-worker-space\\worker-dhn6axcx\r\ndistributed.worker - INFO - -------------------------------------------------\r\ndistributed.worker - INFO - Stopping worker at tcp://127.0.0.1:16061\r\ndistributed.scheduler - INFO - Scheduler closing...\r\ndistributed.scheduler - INFO - Scheduler closing all comms\r\n============================== warnings summary ===============================\r\n```\r\n\r\n</details>"}}
{"actor":{"display_login":"mrocklin"}}