{"action":"opened","issue":{"url":"https://api.github.com/repos/ray-project/ray/issues/1119","repository_url":"https://api.github.com/repos/ray-project/ray","labels_url":"https://api.github.com/repos/ray-project/ray/issues/1119/labels{/name}","comments_url":"https://api.github.com/repos/ray-project/ray/issues/1119/comments","events_url":"https://api.github.com/repos/ray-project/ray/issues/1119/events","html_url":"https://github.com/ray-project/ray/issues/1119","id":265206109,"number":1119,"title":"Ray randomly gives connections error and requires restart","user":{"login":"bdod6","id":20697223,"avatar_url":"https://avatars0.githubusercontent.com/u/20697223?v=4","gravatar_id":"","url":"https://api.github.com/users/bdod6","html_url":"https://github.com/bdod6","followers_url":"https://api.github.com/users/bdod6/followers","following_url":"https://api.github.com/users/bdod6/following{/other_user}","gists_url":"https://api.github.com/users/bdod6/gists{/gist_id}","starred_url":"https://api.github.com/users/bdod6/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bdod6/subscriptions","organizations_url":"https://api.github.com/users/bdod6/orgs","repos_url":"https://api.github.com/users/bdod6/repos","events_url":"https://api.github.com/users/bdod6/events{/privacy}","received_events_url":"https://api.github.com/users/bdod6/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-10-13T08:19:18Z","updated_at":"2017-10-13T08:19:18Z","closed_at":null,"author_association":"NONE","body":"I've been testing Ray on a cluster (deployed through docker/kubernetes) using some tensorflow models in Jupyter notebooks. \r\n\r\nIt seems very unstable though. It will suddenly break drop the connection and require a restart. This happens for trivial calculations, so I don't think it's running out of memory. A line of code will run fine one moment, and then it will disconnect and require restart the next time I run it. This is the error in terminal:\r\n\r\n```\r\n*** Error in `/opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/local_scheduler/../core/src/local_scheduler/local_scheduler': realloc(): invalid pointer: 0x000000000131f7d8 ***\r\n======= Backtrace: =========\r\n/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f62d00447e5]\r\n/lib/x86_64-linux-gnu/libc.so.6(realloc+0x348)[0x7f62d0051a08]\r\n/opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/local_scheduler/../core/src/local_scheduler/local_scheduler[0x44cbbe]\r\n/opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/local_scheduler/../core/src/local_scheduler/local_scheduler[0x44ccfa]\r\n/opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/local_scheduler/../core/src/local_scheduler/local_scheduler[0x44c640]\r\n/opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/local_scheduler/../core/src/local_scheduler/local_scheduler[0x44ea03]\r\n/opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/local_scheduler/../core/src/local_scheduler/local_scheduler[0x44fec8]\r\n/opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/local_scheduler/../core/src/local_scheduler/local_scheduler[0x44ff8a]\r\n/opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/local_scheduler/../core/src/local_scheduler/local_scheduler[0x4275cf]\r\n/opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/local_scheduler/../core/src/local_scheduler/local_scheduler[0x42f73f]\r\n/opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/local_scheduler/../core/src/local_scheduler/local_scheduler[0x430ac0]\r\n/opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/local_scheduler/../core/src/local_scheduler/local_scheduler[0x404752]\r\n/opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/local_scheduler/../core/src/local_scheduler/local_scheduler[0x431577]\r\n/opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/local_scheduler/../core/src/local_scheduler/local_scheduler[0x4317eb]\r\n/opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/local_scheduler/../core/src/local_scheduler/local_scheduler[0x403d24]\r\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f62cffed830]\r\n/opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/local_scheduler/../core/src/local_scheduler/local_scheduler[0x404469]\r\n======= Memory map: ========\r\n00400000-00463000 r-xp 00000000 ca:50 2194438                            /opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/core/src/local_scheduler/local_scheduler\r\n00662000-00663000 r--p 00062000 ca:50 2194438                            /opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/core/src/local_scheduler/local_scheduler\r\n00663000-00664000 rw-p 00063000 ca:50 2194438                            /opt/conda/lib/python2.7/site-packages/ray-0.2.1-py2.7-linux-x86_64.egg/ray/core/src/local_scheduler/local_scheduler\r\n012fd000-01350000 rw-p 00000000 00:00 0                                  [heap]\r\n7f62c8000000-7f62c8021000 rw-p 00000000 00:00 0\r\n7f62c8021000-7f62cc000000 ---p 00000000 00:00 0\r\n7f62cfcc4000-7f62cfdcc000 r-xp 00000000 ca:50 3007292                    /lib/x86_64-linux-gnu/libm-2.23.so\r\n7f62cfdcc000-7f62cffcb000 ---p 00108000 ca:50 3007292                    /lib/x86_64-linux-gnu/libm-2.23.so\r\n7f62cffcb000-7f62cffcc000 r--p 00107000 ca:50 3007292                    /lib/x86_64-linux-gnu/libm-2.23.so\r\n7f62cffcc000-7f62cffcd000 rw-p 00108000 ca:50 3007292                    /lib/x86_64-linux-gnu/libm-2.23.so\r\n7f62cffcd000-7f62d018d000 r-xp 00000000 ca:50 3007260                    /lib/x86_64-linux-gnu/libc-2.23.so\r\n7f62d018d000-7f62d038d000 ---p 001c0000 ca:50 3007260                    /lib/x86_64-linux-gnu/libc-2.23.so\r\n7f62d038d000-7f62d0391000 r--p 001c0000 ca:50 3007260                    /lib/x86_64-linux-gnu/libc-2.23.so\r\n7f62d0391000-7f62d0393000 rw-p 001c4000 ca:50 3007260                    /lib/x86_64-linux-gnu/libc-2.23.so\r\n7f62d0393000-7f62d0397000 rw-p 00000000 00:00 0\r\n7f62d0397000-7f62d03ad000 r-xp 00000000 ca:50 3007281                    /lib/x86_64-linux-gnu/libgcc_s.so.1\r\n7f62d03ad000-7f62d05ac000 ---p 00016000 ca:50 3007281                    /lib/x86_64-linux-gnu/libgcc_s.so.1\r\n7f62d05ac000-7f62d05ad000 rw-p 00015000 ca:50 3007281                    /lib/x86_64-linux-gnu/libgcc_s.so.1\r\n7f62d05ad000-7f62d071f000 r-xp 00000000 ca:50 3008030                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21\r\n7f62d071f000-7f62d091f000 ---p 00172000 ca:50 3008030                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21\r\n7f62d091f000-7f62d0929000 r--p 00172000 ca:50 3008030                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21\r\n7f62d0929000-7f62d092b000 rw-p 0017c000 ca:50 3008030                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21\r\n7f62d092b000-7f62d092f000 rw-p 00000000 00:00 0\r\n7f62d092f000-7f62d0947000 r-xp 00000000 ca:50 3007328                    /lib/x86_64-linux-gnu/libpthread-2.23.so\r\n7f62d0947000-7f62d0b46000 ---p 00018000 ca:50 3007328                    /lib/x86_64-linux-gnu/libpthread-2.23.so\r\n7f62d0b46000-7f62d0b47000 r--p 00017000 ca:50 3007328                    /lib/x86_64-linux-gnu/libpthread-2.23.so\r\n7f62d0b47000-7f62d0b48000 rw-p 00018000 ca:50 3007328                    /lib/x86_64-linux-gnu/libpthread-2.23.so\r\n7f62d0b48000-7f62d0b4c000 rw-p 00000000 00:00 0\r\n7f62d0b4c000-7f62d0b72000 r-xp 00000000 ca:50 3007240                    /lib/x86_64-linux-gnu/ld-2.23.so\r\n7f62d0d64000-7f62d0d6a000 rw-p 00000000 00:00 0\r\n7f62d0d6e000-7f62d0d71000 rw-p 00000000 00:00 0\r\n7f62d0d71000-7f62d0d72000 r--p 00025000 ca:50 3007240                    /lib/x86_64-linux-gnu/ld-2.23.so\r\n7f62d0d72000-7f62d0d73000 rw-p 00026000 ca:50 3007240                    /lib/x86_64-linux-gnu/ld-2.23.so\r\n7f62d0d73000-7f62d0d74000 rw-p 00000000 00:00 0\r\n7ffe4099f000-7ffe409c0000 rw-p 00000000 00:00 0                          [stack]\r\n7ffe409e4000-7ffe409e6000 r--p 00000000 00:00 0                          [vvar]\r\n7ffe409e6000-7ffe409e8000 r-xp 00000000 00:00 0                          [vdso]\r\nffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]\r\n[WARN] (/ray/src/local_scheduler/local_scheduler_client.cc:107) Exiting because local scheduler closed connection.\r\n[WARN] (/ray/src/local_scheduler/local_scheduler_client.cc:107) Exiting because local scheduler closed connection.\r\n[WARN] (/ray/src/local_scheduler/local_scheduler_client.cc:107) Exiting because local scheduler closed connection.\r\n[WARN] (/ray/src/local_scheduler/local_scheduler_client.cc:107) Exiting because local scheduler closed connection.\r\nDisconnecting client on fd 23\r\nDisconnecting client on fd 24\r\nDisconnecting client on fd 21\r\nDisconnecting client on fd 20\r\n[WARN] (/ray/src/local_scheduler/local_scheduler_client.cc:107) Exiting because local scheduler closed connection.\r\nDisconnecting client on fd 19\r\n[WARN] (/ray/src/local_scheduler/local_scheduler_client.cc:107) Exiting because local scheduler closed connection.\r\nDisconnecting client on fd 17\r\n[WARN] (/ray/src/local_scheduler/local_scheduler_client.cc:107) Exiting because local scheduler closed connection.\r\n[WARN] (/ray/src/local_scheduler/local_scheduler_client.cc:107) Exiting because local scheduler closed connection.\r\nDisconnecting client on fd 16\r\nDisconnecting client on fd 14\r\n[WARN] (/ray/src/local_scheduler/local_scheduler_client.cc:107) Exiting because local scheduler closed connection.\r\nDisconnecting client on fd 13\r\nDisconnecting client on fd 9\r\n[WARN] (/ray/src/global_scheduler/global_scheduler.cc:390) Missed too many heartbeats from local scheduler, marking as dead.\r\n```"}}
{"actor":{"display_login":"bdod6"}}