{"action":"created","issue":{"url":"https://api.github.com/repos/Netflix/dynomite/issues/456","repository_url":"https://api.github.com/repos/Netflix/dynomite","labels_url":"https://api.github.com/repos/Netflix/dynomite/issues/456/labels{/name}","comments_url":"https://api.github.com/repos/Netflix/dynomite/issues/456/comments","events_url":"https://api.github.com/repos/Netflix/dynomite/issues/456/events","html_url":"https://github.com/Netflix/dynomite/issues/456","id":237525133,"number":456,"title":"Dynomite immediately dropping connections","user":{"login":"Aatish04","id":29602359,"avatar_url":"https://avatars0.githubusercontent.com/u/29602359?v=3","gravatar_id":"","url":"https://api.github.com/users/Aatish04","html_url":"https://github.com/Aatish04","followers_url":"https://api.github.com/users/Aatish04/followers","following_url":"https://api.github.com/users/Aatish04/following{/other_user}","gists_url":"https://api.github.com/users/Aatish04/gists{/gist_id}","starred_url":"https://api.github.com/users/Aatish04/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Aatish04/subscriptions","organizations_url":"https://api.github.com/users/Aatish04/orgs","repos_url":"https://api.github.com/users/Aatish04/repos","events_url":"https://api.github.com/users/Aatish04/events{/privacy}","received_events_url":"https://api.github.com/users/Aatish04/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-06-21T12:59:04Z","updated_at":"2017-06-27T13:26:55Z","closed_at":null,"body":"I am trying to configured dynomite with two datacenters running on different machine, each will have one rack and one node. Each node connecting to redis instance running on it's machine.\r\nI am trying to start dynomite with gossip enable.\r\nAfter staring dynomite on both the servers after making first request it crashes.\r\n\r\nI am able to connect to redis but not to dynomite\r\n\r\nOn server 1\r\n$ redis-cli -p 8102\r\n127.0.0.1:8102> ping\r\n(error) ERR Storage: Connection timed out\r\n(3.00s)\r\n127.0.0.1:8102>ping\r\nCould not connect to Redis at 127.0.0.1:8102: Connection refused\r\n\r\nOn server 2\r\n$ redis-cli -p 8102\r\n127.0.0.1:8102> ping\r\nError: Connection reset by peer\r\n$ redis-cli -p 8102\r\nCould not connect to Redis at 127.0.0.1:8102: Connection refused\r\n\r\nBelow is my configuration\r\n\r\nServer 1\r\ndyn_o_mite:\r\n  datacenter: dc1\r\n  rack: rack1\r\n  dyn_listen: 0.0.0.0:8101\r\n  dyn_read_timeout: 200000\r\n  dyn_seed_provider: simple_provider\r\n  dyn_seeds:\r\n  - dynomite-host2:8101:rack2:dc2:1383429731\r\n  listen: 0.0.0.0:8102\r\n  preconnect: true\r\n  servers:\r\n  - 127.0.0.1:22122:1\r\n  auto_eject_hosts: true\r\n  server_retry_timeout: 3000\r\n  timeout: 3000\r\n  tokens: '12345678'\r\n  secure_server_option: datacenter\r\n  pem_key_file: conf/dynomite.pem\r\n  data_store: 0\r\n\r\n\r\nServer 2\r\ndyn_o_mite:\r\n  datacenter: dc2\r\n  rack: rack2\r\n  dyn_listen: 0.0.0.0:8101\r\n  dyn_read_timeout: 200000\r\n  dyn_seed_provider: simple_provider\r\n  dyn_seeds:\r\n  - dynomite-host1:8101:rack1:dc1:12345678\r\n  listen: 0.0.0.0:8102\r\n  preconnect: true\r\n  servers:\r\n  - 127.0.0.1:22123:1\r\n  auto_eject_hosts: true\r\n  server_retry_timeout: 3000\r\n  timeout: 3000\r\n  tokens: '1383429731'\r\n  secure_server_option: datacenter\r\n  pem_key_file: conf/dynomite.pem\r\n  data_store: 0\r\n\r\n\r\n\r\nBelow is the log for Server 1\r\n\r\n[2017-06-21 12:50:58.137] conf_validate_pool:2156 setting read_consistency to default value:dc_one\r\n[2017-06-21 12:50:58.137] conf_validate_pool:2163 setting write_consistency to default value:dc_one\r\n[2017-06-21 12:50:58.137] conf_validate_pool:2198 setting env to default value:aws\r\n[2017-06-21 12:50:58.137] conf_validate_pool:2208 setting reconciliation key file to default value:conf/recon_key.pem\r\n[2017-06-21 12:50:58.137] conf_validate_pool:2213 setting reconciliation IV file to default value:conf/recon_iv.pem\r\n[2017-06-21 12:50:58.137] load_private_rsa_key_by_file:68 Private RSA structure filled\r\n[2017-06-21 12:50:58.138] stats_listen:1362 m 5 listening on '0.0.0.0:22222'\r\n[2017-06-21 12:50:58.138] entropy_key_iv_load:364 Key File name: conf/recon_key.pem - IV File name: conf/recon_iv.pem\r\n[2017-06-21 12:50:58.138] entropy_key_iv_load:419 key loaded: 0123456789012345\r\n[2017-06-21 12:50:58.138] entropy_key_iv_load:427 iv loaded: 0123456789012345\r\n[2017-06-21 12:50:58.138] entropy_listen:328 anti-entropy m 8 listening on '127.0.0.1:8105'\r\n[2017-06-21 12:50:58.138] event_base_create:68 e 9 with nevent 1024\r\n[2017-06-21 12:50:58.138] conn_connect:536 connecting to '127.0.0.1:22122:1' on p 10\r\n[2017-06-21 12:50:58.138] proxy_init:124 p 11 listening on '0.0.0.0:8102' in redis pool 'dyn_o_mite'\r\n[2017-06-21 12:50:58.138] dnode_init:112 dyn: p 12 listening on '0.0.0.0:8101' in redis pool 'dyn_o_mite' with 34832512 servers\r\n[2017-06-21 12:50:58.138] preselect_remote_rack_for_replication:1838 my rack index 0\r\n[2017-06-21 12:50:58.138] preselect_remote_rack_for_replication:1865 Selected rack rack2 for replication to remote region dc2\r\n[2017-06-21 12:50:58.138] server_connected:450 connected on s 10 to server '127.0.0.1:22122:1'\r\n[2017-06-21 12:51:15.730] proxy_accept:220 accepted CLIENT 15 on PROXY 11 from '127.0.0.1:50592'\r\n[2017-06-21 12:51:15.731] _msg_get:290 alloc_msg_count: 1 caller: req_get conn: CLIENT sd: 15\r\n[2017-06-21 12:51:15.731] redis_parse_req:1230 parsed unsupported command 'COMMAND'\r\n[2017-06-21 12:51:15.731] redis_parse_req:1829 parsed bad req 1 res 1 type 0 state 5\r\n00000000  2a 31 0d 0a 24 37 0d 0a  43 4f 4d 4d 41 4e 44 0d   |*1..$7..COMMAND.|\r\n00000010  0a                                                 |.|\r\n[2017-06-21 12:51:15.731] core_recv:309 recv on CLIENT 15 failed: Invalid argument\r\n[2017-06-21 12:51:15.731] core_close_log:343 close CLIENT 15 '127.0.0.1:50592' on event 00FF eof 0 done 0 rb 17 sb 0: Invalid argument\r\n[2017-06-21 12:51:15.731] client_close:204 close c 15 discarding pending req 1 len 17 type 0\r\n[2017-06-21 12:51:15.731] client_unref_internal_try_put:101 unref conn 0x2146f30 owner 0x21380a0 from pool 'dyn_o_mite'\r\n[2017-06-21 12:51:17.640] proxy_accept:220 accepted CLIENT 15 on PROXY 11 from '127.0.0.1:50594'\r\n[2017-06-21 12:51:20.641] core_timeout:417 req 2 on SERVER 10 timedout, timeout was 3000\r\n[2017-06-21 12:51:20.641] core_close_log:343 close SERVER 10 '127.0.0.1:22122' on event FF00 eof 0 done 0 rb 0 sb 0: Connection timed out\r\n[2017-06-21 12:51:20.641] _msg_get:290 alloc_msg_count: 2 caller: server_ack_err conn: SERVER sd: 10\r\n[2017-06-21 12:51:20.641] server_ack_err:351 close SERVER 10 req 2:0 len 14 type 86 from c 15: Connection timed out\r\n[2017-06-21 12:51:20.641] client_handle_response:256 ASSERTION FAILED: response 3:0 has peer set\r\n[2017-06-21 12:51:20.641] dn_assert:342 assert '!rsp->peer' failed @ (dyn_client.c, 256)\r\n[2017-06-21 12:51:20.641] msg_local_one_rsp_handler:979 Req 2:0 selected_rsp 3:0\r\n[2017-06-21 12:51:20.641] server_close:405 close SERVER 10 Dropped 0 outqueue & 1 inqueue requests\r\n[2017-06-21 12:51:28.138] gossip_loop:821 I am still joining the ring!\r\n[2017-06-21 12:51:34.256] conn_connect:536 connecting to '127.0.0.1:22122:1' on p 10\r\n[2017-06-21 12:51:34.256] conn_connect:536 connecting to 'dynomite-host2:8101:rack2:dc2:1383429731' on p 16\r\n[2017-06-21 12:51:34.257] dn_stacktrace:326 [0] /lib64/libpthread.so.0() [0x3fb740f7e0]\r\n[2017-06-21 12:51:34.259] dn_stacktrace:326 [1] src/dynomite(mbuf_remove+0x2b) [0x41eb2b]\r\n[2017-06-21 12:51:34.262] dn_stacktrace:326 [2] src/dynomite(dnode_peer_gossip_forward+0x116) [0x416a26]\r\n[2017-06-21 12:51:34.264] dn_stacktrace:326 [3] src/dynomite(dnode_peer_handshake_announcing+0x182) [0x416102]\r\n[2017-06-21 12:51:34.266] dn_stacktrace:326 [4] src/dynomite(core_loop+0x6e) [0x40a26e]\r\n[2017-06-21 12:51:34.268] dn_stacktrace:326 [5] src/dynomite(main+0x6d8) [0x42f338]\r\n[2017-06-21 12:51:34.271] dn_stacktrace:326 [6] /lib64/libc.so.6(__libc_start_main+0xfd) [0x3fb701ed1d]\r\n[2017-06-21 12:51:34.272] dn_stacktrace:326 [7] src/dynomite() [0x4088f9]\r\n[2017-06-21 12:51:34.275] signal_handler:132 signal 11 (SIGSEGV) received, core dumping\r\n\r\n\r\n\r\nbelow is the log for server 2\r\n\r\n[2017-06-21 12:51:10.390] conf_validate_pool:2156 setting read_consistency to default value:dc_one\r\n[2017-06-21 12:51:10.390] conf_validate_pool:2163 setting write_consistency to default value:dc_one\r\n[2017-06-21 12:51:10.390] conf_validate_pool:2198 setting env to default value:aws\r\n[2017-06-21 12:51:10.390] conf_validate_pool:2208 setting reconciliation key file to default value:conf/recon_key.pem\r\n[2017-06-21 12:51:10.390] conf_validate_pool:2213 setting reconciliation IV file to default value:conf/recon_iv.pem\r\n[2017-06-21 12:51:10.390] load_private_rsa_key_by_file:68 Private RSA structure filled\r\n[2017-06-21 12:51:10.390] stats_listen:1362 m 5 listening on '0.0.0.0:22223'\r\n[2017-06-21 12:51:10.390] entropy_key_iv_load:364 Key File name: conf/recon_key.pem - IV File name: conf/recon_iv.pem\r\n[2017-06-21 12:51:10.390] entropy_key_iv_load:419 key loaded: 0123456789012345\r\n[2017-06-21 12:51:10.390] entropy_key_iv_load:427 iv loaded: 0123456789012345\r\n[2017-06-21 12:51:10.390] entropy_listen:328 anti-entropy m 8 listening on '127.0.0.1:8105'\r\n[2017-06-21 12:51:10.391] event_base_create:68 e 9 with nevent 1024\r\n[2017-06-21 12:51:10.391] conn_connect:536 connecting to '127.0.0.1:22123:1' on p 10\r\n[2017-06-21 12:51:10.391] proxy_init:124 p 11 listening on '0.0.0.0:8102' in redis pool 'dyn_o_mite'\r\n[2017-06-21 12:51:10.391] dnode_init:112 dyn: p 12 listening on '0.0.0.0:8101' in redis pool 'dyn_o_mite' with 18210944 servers\r\n[2017-06-21 12:51:10.391] preselect_remote_rack_for_replication:1838 my rack index 0\r\n[2017-06-21 12:51:10.391] preselect_remote_rack_for_replication:1865 Selected rack rack1 for replication to remote region dc1\r\n[2017-06-21 12:51:10.391] server_connected:450 connected on s 10 to server '127.0.0.1:22123:1'\r\n[2017-06-21 12:51:34.247] dnode_accept:172 Accepting client connection from 10.131.12.47/55326 on sd 15\r\n[2017-06-21 12:51:34.247] dnode_accept:217 dyn: accepted LOCAL_PEER_CLIENT 15 on PEER_PROXY 12 from '10.131.12.47:55326'\r\n[2017-06-21 12:51:34.468] _msg_get:290 alloc_msg_count: 1 caller: req_get conn: LOCAL_PEER_CLIENT sd: 15\r\n[2017-06-21 12:51:34.468] conn_recv_data:632 recv on sd 15 eof rb 0 sb 0\r\n[2017-06-21 12:51:34.468] req_recv_next:346 c 15 is done\r\n[2017-06-21 12:51:40.391] gossip_loop:821 I am still joining the ring!\r\n[2017-06-21 12:51:44.040] proxy_accept:220 accepted CLIENT 16 on PROXY 11 from '127.0.0.1:38642'\r\n[2017-06-21 12:51:44.040] conn_connect:536 connecting to 'dynomite-host1:8101:rack1:dc1:12345678' on p 17\r\n[2017-06-21 12:51:44.041] dn_stacktrace:326 [0] /lib64/libpthread.so.0() [0x3fb740f7e0]\r\n[2017-06-21 12:51:44.044] dn_stacktrace:326 [1] src/dynomite(mbuf_remove+0x2b) [0x41eb2b]\r\n[2017-06-21 12:51:44.047] dn_stacktrace:326 [2] src/dynomite(dnode_peer_gossip_forward+0x116) [0x416a26]\r\n[2017-06-21 12:51:44.049] dn_stacktrace:326 [3] src/dynomite(dnode_peer_handshake_announcing+0x182) [0x416102]\r\n[2017-06-21 12:51:44.052] dn_stacktrace:326 [4] src/dynomite(core_loop+0x6e) [0x40a26e]\r\n[2017-06-21 12:51:44.055] dn_stacktrace:326 [5] src/dynomite(main+0x6d8) [0x42f338]\r\n[2017-06-21 12:51:44.057] dn_stacktrace:326 [6] /lib64/libc.so.6(__libc_start_main+0xfd) [0x3fb701ed1d]\r\n[2017-06-21 12:51:44.059] dn_stacktrace:326 [7] src/dynomite() [0x4088f9]\r\n[2017-06-21 12:51:44.061] signal_handler:132 signal 11 (SIGSEGV) received, core dumping\r\n\r\n\r\nLooks like gossiping not working but both hosts are accessible to each other.\r\nI am using master branch.\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/Netflix/dynomite/issues/comments/311357476","html_url":"https://github.com/Netflix/dynomite/issues/456#issuecomment-311357476","issue_url":"https://api.github.com/repos/Netflix/dynomite/issues/456","id":311357476,"user":{"login":"ipapapa","id":4562887,"avatar_url":"https://avatars3.githubusercontent.com/u/4562887?v=3","gravatar_id":"","url":"https://api.github.com/users/ipapapa","html_url":"https://github.com/ipapapa","followers_url":"https://api.github.com/users/ipapapa/followers","following_url":"https://api.github.com/users/ipapapa/following{/other_user}","gists_url":"https://api.github.com/users/ipapapa/gists{/gist_id}","starred_url":"https://api.github.com/users/ipapapa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ipapapa/subscriptions","organizations_url":"https://api.github.com/users/ipapapa/orgs","repos_url":"https://api.github.com/users/ipapapa/repos","events_url":"https://api.github.com/users/ipapapa/events{/privacy}","received_events_url":"https://api.github.com/users/ipapapa/received_events","type":"User","site_admin":false},"created_at":"2017-06-27T13:26:55Z","updated_at":"2017-06-27T13:26:55Z","body":"Any Redis version is compatible... The problem is that the new `redis-cli` has an extra command for which we need to add support for. We use 3.0.7 in our systems. I will let @shailesh33 comment what might have caused the segmentation fault. I see the following assert statement:\r\n> [2017-06-27 06:22:44.161] dn_assert:342 assert 'req->done && !req->swallow' failed @ (dyn_dnode_client.c, 531)\r\n"}}
{"actor":{"display_login":"ipapapa"}}