{"action":"created","issue":{"url":"https://api.github.com/repos/arangodb/arangodb/issues/2480","repository_url":"https://api.github.com/repos/arangodb/arangodb","labels_url":"https://api.github.com/repos/arangodb/arangodb/issues/2480/labels{/name}","comments_url":"https://api.github.com/repos/arangodb/arangodb/issues/2480/comments","events_url":"https://api.github.com/repos/arangodb/arangodb/issues/2480/events","html_url":"https://github.com/arangodb/arangodb/issues/2480","id":228463039,"number":2480,"title":"Cluster is not stable for big data: dbserver can't register to agency after cluster restart","user":{"login":"itoldman","id":4068895,"avatar_url":"https://avatars2.githubusercontent.com/u/4068895?v=3","gravatar_id":"","url":"https://api.github.com/users/itoldman","html_url":"https://github.com/itoldman","followers_url":"https://api.github.com/users/itoldman/followers","following_url":"https://api.github.com/users/itoldman/following{/other_user}","gists_url":"https://api.github.com/users/itoldman/gists{/gist_id}","starred_url":"https://api.github.com/users/itoldman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/itoldman/subscriptions","organizations_url":"https://api.github.com/users/itoldman/orgs","repos_url":"https://api.github.com/users/itoldman/repos","events_url":"https://api.github.com/users/itoldman/events{/privacy}","received_events_url":"https://api.github.com/users/itoldman/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-05-13T09:59:38Z","updated_at":"2017-05-13T11:52:41Z","closed_at":null,"body":"## my environment running ArangoDB\r\nI'm using the latest ArangoDB of the respective release series:\r\n- [ ] 2.8\r\n- [ ] 3.0\r\n- [ ] 3.1\r\n- [* ] 3.2 pre-releases: ___\r\n- [ ] self-compiled devel branch\r\n\r\nOn this operating system:\r\n- [ ] DCOS on\r\n  - [ ] AWS\r\n  - [ ] Azure\r\n  - [ ] own infrastructure\r\n- [ ] Linux \r\n  - [ ] Debian .deb\r\n  - [ ] Ubuntu .deb\r\n  - [ ] SUSE   .rpm\r\n  - [ ] RedHat .rpm\r\n  - [ *] Fedora .rpm\r\n  - [ ] Gentoo\r\n  - [ ] docker - official docker library\r\n  - [ ] other:\r\n- [ ] Windows, version: \r\n- [ ] MacOS, version:\r\n\r\nI'm testing cluster is stable or not with rocksdb engine. I used arangodb tool build a cluster with three nodes. Then I insterted many data. Then I shutdown all of three servers. Then I restarted three servers. I expected the cluster restarted successfully. But only three agency servers started successfully. All of db servers and coordinator servers reported can't start after 5 mins. From logs, it reported Fatal can't register to agency. I'm wondering what's the process of cluster restart and how  it restore the big data and the cluster state."},"comment":{"url":"https://api.github.com/repos/arangodb/arangodb/issues/comments/301243475","html_url":"https://github.com/arangodb/arangodb/issues/2480#issuecomment-301243475","issue_url":"https://api.github.com/repos/arangodb/arangodb/issues/2480","id":301243475,"user":{"login":"itoldman","id":4068895,"avatar_url":"https://avatars2.githubusercontent.com/u/4068895?v=3","gravatar_id":"","url":"https://api.github.com/users/itoldman","html_url":"https://github.com/itoldman","followers_url":"https://api.github.com/users/itoldman/followers","following_url":"https://api.github.com/users/itoldman/following{/other_user}","gists_url":"https://api.github.com/users/itoldman/gists{/gist_id}","starred_url":"https://api.github.com/users/itoldman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/itoldman/subscriptions","organizations_url":"https://api.github.com/users/itoldman/orgs","repos_url":"https://api.github.com/users/itoldman/repos","events_url":"https://api.github.com/users/itoldman/events{/privacy}","received_events_url":"https://api.github.com/users/itoldman/received_events","type":"User","site_admin":false},"created_at":"2017-05-13T11:52:41Z","updated_at":"2017-05-13T11:52:41Z","body":"[root@localhost ~]# nohup /root/arangodb --dataDir /home/gaxz/ --arangod=/root/arangodbsource/build/bin/arangod --jsDir=/root/arangodbsource/js --log.verbose=true &\r\n[1] 14438\r\n[root@localhost ~]# nohup: 忽略输入并把输出追加到\"nohup.out\"\r\n\r\n[root@localhost ~]# tail -f nohup.out\r\n2017/05/13 14:54:37 Starting agent on port 8531\r\n2017/05/13 14:54:37 Waiting on 14445\r\n2017/05/13 14:54:38 Looking for a running instance of dbserver on port 8530\r\n2017/05/13 14:54:38 Cannot signal process 6151\r\n2017/05/13 14:54:38 Starting dbserver on port 8530\r\n2017/05/13 14:54:38 Waiting on 14489\r\n2017/05/13 14:54:39 Looking for a running instance of coordinator on port 8529\r\n2017/05/13 14:54:39 Cannot find /home/gaxz/coordinator8529/data/LOCK\r\n2017/05/13 14:54:39 Starting coordinator on port 8529\r\n2017/05/13 14:54:39 Waiting on 14493\r\n\r\n2017/05/13 14:56:37 agent up and running (version 3.2.milestone3).\r\n2017/05/13 14:57:08 dbserver not ready after 5min!\r\n2017/05/13 14:57:09 coordinator not ready after 5min!\r\n2017/05/13 14:57:16 Wait on 14489 returned <nil>\r\n2017/05/13 14:57:16 dbserver has terminated\r\n2017/05/13 14:57:16 restarting dbserver\r\n2017/05/13 14:57:16 Looking for a running instance of dbserver on port 8530\r\n2017/05/13 14:57:16 Cannot signal process 6151\r\n2017/05/13 14:57:16 Starting dbserver on port 8530\r\n2017/05/13 14:57:16 Waiting on 14516\r\n2017/05/13 14:57:38 Wait on 14516 returned <nil>\r\n2017/05/13 14:57:38 dbserver has terminated, quickly, in 22.073555641s (recent failures: 1)\r\n2017/05/13 14:57:38 restarting dbserver\r\n2017/05/13 14:57:38 Looking for a running instance of dbserver on port 8530\r\n2017/05/13 14:57:38 Cannot signal process 6151\r\n2017/05/13 14:57:38 Starting dbserver on port 8530\r\n2017/05/13 14:57:38 Waiting on 14526\r\n2017/05/13 14:57:40 Wait on 14526 returned <nil>\r\n2017/05/13 14:57:40 dbserver has terminated, quickly, in 1.259396713s (recent failures: 2)\r\n2017/05/13 14:57:40 ## Start of dbserver log\r\n\t2017-05-13T06:57:38Z [14526] DEBUG connecting to ip endpoint 'http+tcp://192.168.130.201:8531'\r\n\t2017-05-13T06:57:38Z [14526] TRACE bind to address '192.168.130.201', port 8531\r\n\t2017-05-13T06:57:38Z [14526] TRACE Request: POST /_api/agency/write HTTP/1.1\\r\\nHost: 192.168.130.202\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 134\\r\\n\\r\\n[[{\"arango/InitDone\":{\"new\":false,\"op\":\"set\",\"ttl\":60}},{\"arango/InitDone\":{\"oldEmpty\":true}},\"302719aa-f6a3-4645-97bf-02828f54cadd\"]]\r\n\t2017-05-13T06:57:38Z [14526] DEBUG connecting to ip endpoint 'http+tcp://192.168.130.202:8531'\r\n\t2017-05-13T06:57:38Z [14526] TRACE bind to address '192.168.130.202', port 8531\r\n\t2017-05-13T06:57:38Z [14526] TRACE Request: POST /_api/agency/inquire HTTP/1.1\\r\\nHost: 192.168.130.202\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 40\\r\\n\\r\\n[\"302719aa-f6a3-4645-97bf-02828f54cadd\"]\r\n\t2017-05-13T06:57:38Z [14526] TRACE Request: POST /_api/agency/read HTTP/1.1\\r\\nHost: 192.168.130.202\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 21\\r\\n\\r\\n[[\"arango/InitDone\"]]\r\n\t2017-05-13T06:57:38Z [14526] DEBUG connecting to ip endpoint 'http+tcp://192.168.130.202:8531'\r\n\t2017-05-13T06:57:38Z [14526] TRACE bind to address '192.168.130.202', port 8531\r\n\t2017-05-13T06:57:38Z [14526] TRACE Request: POST /_api/agency/read HTTP/1.1\\r\\nHost: 192.168.130.202\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 19\\r\\n\\r\\n[[\"arango/Secret\"]]\r\n\t2017-05-13T06:57:38Z [14526] TRACE Request: POST /_api/agency/read HTTP/1.1\\r\\nHost: 192.168.130.202\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 27\\r\\n\\r\\n[[\"arango/Plan/DBServers\"]]\r\n\t2017-05-13T06:57:38Z [14526] TRACE Request: POST /_api/agency/transact HTTP/1.1\\r\\nHost: 192.168.130.202\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 377\\r\\n\\r\\n[[{\"arango/Plan/DBServers/PRMR-4c923f39-5b15-41f5-b8f0-3ed5bf6f7210\":{\"new\":\"none\",\"op\":\"set\"}},{\"arango/Plan/DBServers/PRMR-4c923f39-5b15-41f5-b8f0-3ed5bf6f7210\":{\"oldEmpty\":true}},\"\"],[{\"arango/Current/DBServers/PRMR-4c923f39-5b15-41f5-b8f0-3ed5bf6f7210\":{\"new\":\"none\",\"op\":\"set\"}},{\"arango/Current/DBServers/PRMR-4c923f39-5b15-41f5-b8f0-3ed5bf6f7210\":{\"oldEmpty\":true}},\"\"]]\r\n\t2017-05-13T06:57:38Z [14526] TRACE Request: POST /_api/agency/read HTTP/1.1\\r\\nHost: 192.168.130.202\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 113\\r\\n\\r\\n[[\"arango/Target/LatestDBServerId\",\"arango/Target/MapUniqueToShortID/PRMR-4c923f39-5b15-41f5-b8f0-3ed5bf6f7210\"]]\r\n\t2017-05-13T06:57:38Z [14526] TRACE Request: POST /_api/agency/read HTTP/1.1\\r\\nHost: 192.168.130.202\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 30\\r\\n\\r\\n[[\"arango/Plan/Coordinators\"]]\r\n\t2017-05-13T06:57:38Z [14526] TRACE Request: POST /_api/agency/read HTTP/1.1\\r\\nHost: 192.168.130.202\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 27\\r\\n\\r\\n[[\"arango/Plan/DBServers\"]]\r\n\t2017-05-13T06:57:38Z [14526] TRACE Request: POST /_api/agency/write HTTP/1.1\\r\\nHost: 192.168.130.202\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 142\\r\\n\\r\\n[[{\"arango/Current/DBServers/PRMR-4c923f39-5b15-41f5-b8f0-3ed5bf6f7210\":{\"new\":\"none\",\"op\":\"set\"}},{},\"eb101e91-b8d9-4012-8cee-e8b2183cfc66\"]]\r\n\t2017-05-13T06:57:39Z [14526] TRACE Request: POST /_api/agency/inquire HTTP/1.1\\r\\nHost: 192.168.130.202\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 40\\r\\n\\r\\n[\"eb101e91-b8d9-4012-8cee-e8b2183cfc66\"]\r\n\t2017-05-13T06:57:39Z [14526] DEBUG connecting to ip endpoint 'http+tcp://192.168.130.202:8531'\r\n\t2017-05-13T06:57:39Z [14526] TRACE bind to address '192.168.130.202', port 8531\r\n\t2017-05-13T06:57:39Z [14526] FATAL unable to register server in agency\r\n2017/05/13 14:57:40 ## End of dbserver log\r\n2017/05/13 14:57:40 restarting dbserver\r\n2017/05/13 14:57:40 Looking for a running instance of dbserver on port 8530\r\n2017/05/13 14:57:40 Cannot signal process 6151\r\n2017/05/13 14:57:40 Starting dbserver on port 8530\r\n2017/05/13 14:57:40 Waiting on 14529\r\n2017/05/13 14:57:56 Wait on 14529 returned <nil>\r\n2017/05/13 14:57:56 dbserver has terminated, quickly, in 16.002287214s (recent failures: 3)\r\n\r\n2017/05/13 14:58:02 coordinator has terminated, quickly, in 5.880883661s (recent failures: 1)\r\n2017/05/13 14:58:02 restarting coordinator\r\n2017/05/13 14:58:02 Looking for a running instance of coordinator on port 8529\r\n2017/05/13 14:58:02 Cannot find /home/gaxz/coordinator8529/data/LOCK\r\n2017/05/13 14:58:02 Starting coordinator on port 8529\r\n2017/05/13 14:58:02 Waiting on 14548\r\n2017/05/13 14:58:03 Wait on 14548 returned <nil>\r\n2017/05/13 14:58:03 coordinator has terminated, quickly, in 1.16431161s (recent failures: 2)\r\n2017/05/13 14:58:03 ## Start of coordinator log\r\n\t2017-05-13T06:58:02Z [14548] DEBUG file-descriptors (nofiles) hard limit is 999999, soft limit is 999999\r\n\t2017-05-13T06:58:02Z [14548] TRACE page size is 4096\r\n\t2017-05-13T06:58:02Z [14548] TRACE shutdown file found\r\n\t2017-05-13T06:58:02Z [14548] TRACE previous shutdown was at '2017-05-13T05:27:26Z'\r\n\t2017-05-13T06:58:02Z [14548] TRACE initial values for WAL logfile manager: tick: 61703865, hlc: 1567257522819039232, lastCollected: 0, lastSealed: 0\r\n\t2017-05-13T06:58:02Z [14548] INFO Starting up with role COORDINATOR\r\n\t2017-05-13T06:58:02Z [14548] TRACE Request: POST /_api/agency/write HTTP/1.1\\r\\nHost: 192.168.130.201\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 134\\r\\n\\r\\n[[{\"arango/InitDone\":{\"new\":false,\"op\":\"set\",\"ttl\":60}},{\"arango/InitDone\":{\"oldEmpty\":true}},\"67bbe373-a048-4b63-9554-3e92954eeb54\"]]\r\n\t2017-05-13T06:58:02Z [14548] DEBUG connecting to ip endpoint 'http+tcp://192.168.130.201:8531'\r\n\t2017-05-13T06:58:02Z [14548] TRACE bind to address '192.168.130.201', port 8531\r\n\t2017-05-13T06:58:02Z [14548] TRACE Request: POST /_api/agency/read HTTP/1.1\\r\\nHost: 192.168.130.201\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 21\\r\\n\\r\\n[[\"arango/InitDone\"]]\r\n\t2017-05-13T06:58:02Z [14548] TRACE Request: POST /_api/agency/read HTTP/1.1\\r\\nHost: 192.168.130.201\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 19\\r\\n\\r\\n[[\"arango/Secret\"]]\r\n\t2017-05-13T06:58:02Z [14548] TRACE Request: POST /_api/agency/read HTTP/1.1\\r\\nHost: 192.168.130.201\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 30\\r\\n\\r\\n[[\"arango/Plan/Coordinators\"]]\r\n\t2017-05-13T06:58:02Z [14548] TRACE Request: POST /_api/agency/transact HTTP/1.1\\r\\nHost: 192.168.130.201\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 389\\r\\n\\r\\n[[{\"arango/Plan/Coordinators/CRDN-d350105d-335c-4f9c-ba2c-651253806ef4\":{\"new\":\"none\",\"op\":\"set\"}},{\"arango/Plan/Coordinators/CRDN-d350105d-335c-4f9c-ba2c-651253806ef4\":{\"oldEmpty\":true}},\"\"],[{\"arango/Current/Coordinators/CRDN-d350105d-335c-4f9c-ba2c-651253806ef4\":{\"new\":\"none\",\"op\":\"set\"}},{\"arango/Current/Coordinators/CRDN-d350105d-335c-4f9c-ba2c-651253806ef4\":{\"oldEmpty\":true}},\"\"]]\r\n\t2017-05-13T06:58:02Z [14548] TRACE Request: POST /_api/agency/read HTTP/1.1\\r\\nHost: 192.168.130.201\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 116\\r\\n\\r\\n[[\"arango/Target/LatestCoordinatorId\",\"arango/Target/MapUniqueToShortID/CRDN-d350105d-335c-4f9c-ba2c-651253806ef4\"]]\r\n\t2017-05-13T06:58:02Z [14548] TRACE Request: POST /_api/agency/read HTTP/1.1\\r\\nHost: 192.168.130.201\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 30\\r\\n\\r\\n[[\"arango/Plan/Coordinators\"]]\r\n\t2017-05-13T06:58:02Z [14548] TRACE Request: POST /_api/agency/write HTTP/1.1\\r\\nHost: 192.168.130.201\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 145\\r\\n\\r\\n[[{\"arango/Current/Coordinators/CRDN-d350105d-335c-4f9c-ba2c-651253806ef4\":{\"new\":\"none\",\"op\":\"set\"}},{},\"c8d09d66-71e8-4d73-930b-2b73026288d4\"]]\r\n\t2017-05-13T06:58:03Z [14548] TRACE Request: POST /_api/agency/inquire HTTP/1.1\\r\\nHost: 192.168.130.201\\r\\nConnection: Keep-Alive\\r\\nUser-Agent: ArangoDB\\r\\nAccept-Encoding: deflate\\r\\ncontent-type: application/json\\r\\nContent-Length: 40\\r\\n\\r\\n[\"c8d09d66-71e8-4d73-930b-2b73026288d4\"]\r\n\t2017-05-13T06:58:03Z [14548] DEBUG connecting to ip endpoint 'http+tcp://192.168.130.201:8531'\r\n\t2017-05-13T06:58:03Z [14548] TRACE bind to address '192.168.130.201', port 8531\r\n\t2017-05-13T06:58:03Z [14548] FATAL unable to register server in agency\r\n2017/05/13 14:58:03 ## End of coordinator log\r\n"}}
{"actor":{"display_login":"itoldman"}}