{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd-operator/issues/1121","repository_url":"https://api.github.com/repos/coreos/etcd-operator","labels_url":"https://api.github.com/repos/coreos/etcd-operator/issues/1121/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd-operator/issues/1121/comments","events_url":"https://api.github.com/repos/coreos/etcd-operator/issues/1121/events","html_url":"https://github.com/coreos/etcd-operator/issues/1121","id":230607124,"number":1121,"title":"Endless creation of members","user":{"login":"ravindranathakila","id":1250044,"avatar_url":"https://avatars0.githubusercontent.com/u/1250044?v=3","gravatar_id":"","url":"https://api.github.com/users/ravindranathakila","html_url":"https://github.com/ravindranathakila","followers_url":"https://api.github.com/users/ravindranathakila/followers","following_url":"https://api.github.com/users/ravindranathakila/following{/other_user}","gists_url":"https://api.github.com/users/ravindranathakila/gists{/gist_id}","starred_url":"https://api.github.com/users/ravindranathakila/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ravindranathakila/subscriptions","organizations_url":"https://api.github.com/users/ravindranathakila/orgs","repos_url":"https://api.github.com/users/ravindranathakila/repos","events_url":"https://api.github.com/users/ravindranathakila/events{/privacy}","received_events_url":"https://api.github.com/users/ravindranathakila/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-05-23T07:09:25Z","updated_at":"2017-05-23T22:30:04Z","closed_at":null,"body":"Two members got created and the rest are in an endless creation state:\r\n\r\netcd-operator:\r\n\r\n```\r\n\r\n2017-05-23T07:01:46.786271705Z time=\"2017-05-23T07:01:46Z\" level=info msg=\"running members: etcd-cluster-0007,etcd-cluster-0006\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:01:46.786339931Z time=\"2017-05-23T07:01:46Z\" level=info msg=\"cluster membership: etcd-cluster-0006,etcd-cluster-0007\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:01:46.846388637Z time=\"2017-05-23T07:01:46Z\" level=info msg=\"added member (etcd-cluster-0014)\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:01:46.846667760Z time=\"2017-05-23T07:01:46Z\" level=info msg=\"Finish reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:01:55.100072678Z time=\"2017-05-23T07:01:55Z\" level=info msg=\"Start reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:01:55.100356782Z time=\"2017-05-23T07:01:55Z\" level=info msg=\"running members: etcd-cluster-0006,etcd-cluster-0007,etcd-cluster-0014\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:01:55.185855558Z time=\"2017-05-23T07:01:55Z\" level=info msg=\"cluster membership: etcd-cluster-0006,etcd-cluster-0007,etcd-cluster-0014\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:01:55.191540354Z time=\"2017-05-23T07:01:55Z\" level=error msg=\"fail to add new member (etcd-cluster-0015): etcdserver: re-configuration failed due to not enough started members\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:01:55.191756908Z time=\"2017-05-23T07:01:55Z\" level=info msg=\"Finish reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:01:55.191822532Z time=\"2017-05-23T07:01:55Z\" level=error msg=\"failed to reconcile: etcdserver: re-configuration failed due to not enough started members\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:03.291971894Z time=\"2017-05-23T07:02:03Z\" level=info msg=\"Start reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:03.292066470Z time=\"2017-05-23T07:02:03Z\" level=info msg=\"running members: etcd-cluster-0006,etcd-cluster-0007\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:03.292125681Z time=\"2017-05-23T07:02:03Z\" level=info msg=\"cluster membership: etcd-cluster-0014,etcd-cluster-0006,etcd-cluster-0007\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:03.292190023Z time=\"2017-05-23T07:02:03Z\" level=info msg=\"removing one dead member\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:03.292245009Z time=\"2017-05-23T07:02:03Z\" level=info msg=\"removing dead member \\\"etcd-cluster-0014\\\"\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:03.347459902Z time=\"2017-05-23T07:02:03Z\" level=info msg=\"removed member (etcd-cluster-0014) with ID (513064720475007206)\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:03.347558575Z time=\"2017-05-23T07:02:03Z\" level=info msg=\"Finish reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:11.375568972Z time=\"2017-05-23T07:02:11Z\" level=info msg=\"Start reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:11.375697682Z time=\"2017-05-23T07:02:11Z\" level=info msg=\"running members: etcd-cluster-0006,etcd-cluster-0007\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:11.375857324Z time=\"2017-05-23T07:02:11Z\" level=info msg=\"cluster membership: etcd-cluster-0006,etcd-cluster-0007\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:11.432989179Z time=\"2017-05-23T07:02:11Z\" level=info msg=\"added member (etcd-cluster-0015)\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:11.433241113Z time=\"2017-05-23T07:02:11Z\" level=info msg=\"Finish reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:19.476055266Z time=\"2017-05-23T07:02:19Z\" level=info msg=\"Start reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:19.476176644Z time=\"2017-05-23T07:02:19Z\" level=info msg=\"running members: etcd-cluster-0006,etcd-cluster-0007,etcd-cluster-0015\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:19.485842138Z time=\"2017-05-23T07:02:19Z\" level=info msg=\"cluster membership: etcd-cluster-0015,etcd-cluster-0006,etcd-cluster-0007\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:19.495124262Z time=\"2017-05-23T07:02:19Z\" level=error msg=\"fail to add new member (etcd-cluster-0016): etcdserver: re-configuration failed due to not enough started members\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:19.495138120Z time=\"2017-05-23T07:02:19Z\" level=info msg=\"Finish reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:19.495143135Z time=\"2017-05-23T07:02:19Z\" level=error msg=\"failed to reconcile: etcdserver: re-configuration failed due to not enough started members\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:27.590439077Z time=\"2017-05-23T07:02:27Z\" level=info msg=\"Start reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:27.590537436Z time=\"2017-05-23T07:02:27Z\" level=info msg=\"running members: etcd-cluster-0006,etcd-cluster-0007\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:27.590610095Z time=\"2017-05-23T07:02:27Z\" level=info msg=\"cluster membership: etcd-cluster-0015,etcd-cluster-0006,etcd-cluster-0007\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:27.590660040Z time=\"2017-05-23T07:02:27Z\" level=info msg=\"removing one dead member\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:27.590692589Z time=\"2017-05-23T07:02:27Z\" level=info msg=\"removing dead member \\\"etcd-cluster-0015\\\"\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:27.642457320Z time=\"2017-05-23T07:02:27Z\" level=info msg=\"removed member (etcd-cluster-0015) with ID (3149357175879294987)\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:27.642553528Z time=\"2017-05-23T07:02:27Z\" level=info msg=\"Finish reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:35.703405241Z time=\"2017-05-23T07:02:35Z\" level=info msg=\"Start reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:35.703492652Z time=\"2017-05-23T07:02:35Z\" level=info msg=\"running members: etcd-cluster-0006,etcd-cluster-0007\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:35.785856443Z time=\"2017-05-23T07:02:35Z\" level=info msg=\"cluster membership: etcd-cluster-0006,etcd-cluster-0007\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:35.809909646Z time=\"2017-05-23T07:02:35Z\" level=info msg=\"added member (etcd-cluster-0016)\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:35.810125592Z time=\"2017-05-23T07:02:35Z\" level=info msg=\"Finish reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:43.840585160Z time=\"2017-05-23T07:02:43Z\" level=info msg=\"Start reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:43.840881652Z time=\"2017-05-23T07:02:43Z\" level=info msg=\"running members: etcd-cluster-0016,etcd-cluster-0006,etcd-cluster-0007\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:43.885805494Z time=\"2017-05-23T07:02:43Z\" level=info msg=\"cluster membership: etcd-cluster-0016,etcd-cluster-0006,etcd-cluster-0007\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:43.893966515Z time=\"2017-05-23T07:02:43Z\" level=error msg=\"fail to add new member (etcd-cluster-0017): etcdserver: re-configuration failed due to not enough started members\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:43.894151425Z time=\"2017-05-23T07:02:43Z\" level=info msg=\"Finish reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:43.894212435Z time=\"2017-05-23T07:02:43Z\" level=error msg=\"failed to reconcile: etcdserver: re-configuration failed due to not enough started members\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:51.993039041Z time=\"2017-05-23T07:02:51Z\" level=info msg=\"Start reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:51.993129145Z time=\"2017-05-23T07:02:51Z\" level=info msg=\"running members: etcd-cluster-0006,etcd-cluster-0007\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:51.993185753Z time=\"2017-05-23T07:02:51Z\" level=info msg=\"cluster membership: etcd-cluster-0006,etcd-cluster-0007,etcd-cluster-0016\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:51.993217619Z time=\"2017-05-23T07:02:51Z\" level=info msg=\"removing one dead member\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:51.993269531Z time=\"2017-05-23T07:02:51Z\" level=info msg=\"removing dead member \\\"etcd-cluster-0016\\\"\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:52.080450417Z time=\"2017-05-23T07:02:52Z\" level=info msg=\"removed member (etcd-cluster-0016) with ID (13280956518315516633)\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:02:52.080465927Z time=\"2017-05-23T07:02:52Z\" level=info msg=\"Finish reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:03:00.110650447Z time=\"2017-05-23T07:03:00Z\" level=info msg=\"Start reconciling\" cluster-name=etcd-cluster pkg=cluster \r\n2017-05-23T07:03:00.110877583Z time=\"2017-05-23T07:03:00Z\" level=info msg=\"running members: etcd-cluster-0007,etcd-cluster-0006\" cluster-name=etcd-cluster pkg=cluster \r\n```\r\n\r\n\r\n006 Logs:\r\n\r\n```\r\n2017-05-23T07:03:48.994932858Z 2017-05-23 07:03:48.994911 I | rafthttp: starting peer d7cea5d10f262f59...\r\n2017-05-23T07:03:48.995002048Z 2017-05-23 07:03:48.994983 I | rafthttp: started HTTP pipelining with peer d7cea5d10f262f59\r\n2017-05-23T07:03:48.998697221Z 2017-05-23 07:03:48.998630 I | rafthttp: started peer d7cea5d10f262f59\r\n2017-05-23T07:03:48.999418606Z 2017-05-23 07:03:48.999384 I | rafthttp: added peer d7cea5d10f262f59\r\n2017-05-23T07:03:49.000648796Z 2017-05-23 07:03:49.000601 I | rafthttp: started streaming with peer d7cea5d10f262f59 (writer)\r\n2017-05-23T07:03:49.000847927Z 2017-05-23 07:03:49.000681 I | rafthttp: started streaming with peer d7cea5d10f262f59 (writer)\r\n2017-05-23T07:03:49.000957697Z 2017-05-23 07:03:49.000920 I | rafthttp: started streaming with peer d7cea5d10f262f59 (stream MsgApp v2 reader)\r\n2017-05-23T07:03:49.003109413Z 2017-05-23 07:03:49.003068 I | rafthttp: started streaming with peer d7cea5d10f262f59 (stream Message reader)\r\n2017-05-23T07:03:52.831557900Z 2017-05-23 07:03:52.831479 W | etcdserver: failed to reach the peerURL(http://etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local:2380) of member d7cea5d10f262f59 (Get http://etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local:2380/version: dial tcp 10.4.9.45:2380: getsockopt: connection refused)\r\n2017-05-23T07:03:52.831639309Z 2017-05-23 07:03:52.831598 W | etcdserver: cannot get the version of member d7cea5d10f262f59 (Get http://etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local:2380/version: dial tcp 10.4.9.45:2380: getsockopt: connection refused)\r\n2017-05-23T07:03:54.003562223Z 2017-05-23 07:03:54.003442 W | rafthttp: health check for peer d7cea5d10f262f59 could not connect: dial tcp 10.4.9.45:2380: getsockopt: connection refused\r\n2017-05-23T07:03:56.968727939Z 2017-05-23 07:03:56.968619 W | etcdserver: failed to reach the peerURL(http://etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local:2380) of member d7cea5d10f262f59 (Get http://etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local:2380/version: read tcp 10.4.5.177:46408->10.4.9.45:2380: read: connection reset by peer)\r\n2017-05-23T07:03:56.968834537Z 2017-05-23 07:03:56.968812 W | etcdserver: cannot get the version of member d7cea5d10f262f59 (Get http://etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local:2380/version: read tcp 10.4.5.177:46408->10.4.9.45:2380: read: connection reset by peer)\r\n2017-05-23T07:03:59.010784616Z 2017-05-23 07:03:59.010666 W | rafthttp: health check for peer d7cea5d10f262f59 could not connect: dial tcp 10.4.9.45:2380: getsockopt: connection refused\r\n2017-05-23T07:04:00.979546267Z 2017-05-23 07:04:00.979427 W | etcdserver: failed to reach the peerURL(http://etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local:2380) of member d7cea5d10f262f59 (Get http://etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local:2380/version: dial tcp: lookup etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local on 10.7.240.10:53: no such host)\r\n2017-05-23T07:04:00.979818070Z 2017-05-23 07:04:00.979790 W | etcdserver: cannot get the version of member d7cea5d10f262f59 (Get http://etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local:2380/version: dial tcp: lookup etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local on 10.7.240.10:53: no such host)\r\n2017-05-23T07:04:04.016892139Z 2017-05-23 07:04:04.016780 W | rafthttp: health check for peer d7cea5d10f262f59 could not connect: dial tcp: lookup etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local on 10.7.240.10:53: no such host\r\n2017-05-23T07:04:04.989497357Z 2017-05-23 07:04:04.989388 W | etcdserver: failed to reach the peerURL(http://etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local:2380) of member d7cea5d10f262f59 (Get http://etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local:2380/version: dial tcp: lookup etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local on 10.7.240.10:53: no such host)\r\n2017-05-23T07:04:04.989740210Z 2017-05-23 07:04:04.989701 W | etcdserver: cannot get the version of member d7cea5d10f262f59 (Get http://etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local:2380/version: dial tcp: lookup etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local on 10.7.240.10:53: no such host)\r\n2017-05-23T07:04:05.211484424Z 2017-05-23 07:04:05.211375 I | etcdserver/membership: removed member d7cea5d10f262f59 from cluster af539b1f41ca4937\r\n2017-05-23T07:04:05.211594167Z 2017-05-23 07:04:05.211573 I | rafthttp: stopping peer d7cea5d10f262f59...\r\n2017-05-23T07:04:05.211821861Z 2017-05-23 07:04:05.211799 I | rafthttp: stopped streaming with peer d7cea5d10f262f59 (writer)\r\n2017-05-23T07:04:05.211881107Z 2017-05-23 07:04:05.211863 I | rafthttp: stopped streaming with peer d7cea5d10f262f59 (writer)\r\n2017-05-23T07:04:05.215903097Z 2017-05-23 07:04:05.215835 I | rafthttp: stopped HTTP pipelining with peer d7cea5d10f262f59\r\n2017-05-23T07:04:05.217433975Z 2017-05-23 07:04:05.217390 I | rafthttp: stopped streaming with peer d7cea5d10f262f59 (stream MsgApp v2 reader)\r\n2017-05-23T07:04:05.217766645Z 2017-05-23 07:04:05.217735 I | rafthttp: stopped streaming with peer d7cea5d10f262f59 (stream Message reader)\r\n2017-05-23T07:04:05.218099241Z 2017-05-23 07:04:05.218069 I | rafthttp: stopped peer d7cea5d10f262f59\r\n2017-05-23T07:04:05.218259282Z 2017-05-23 07:04:05.218227 I | rafthttp: removed peer d7cea5d10f262f59\r\n2017-05-23T07:04:13.296839537Z 2017-05-23 07:04:13.296718 I | etcdserver/membership: added member 926640b1cf5e96bb [http://etcd-cluster-0020.etcd-cluster.pump.svc.cluster.local:2380] to cluster af539b1f41ca4937\r\n2017-05-23T07:04:13.298499753Z 2017-05-23 07:04:13.298396 I | rafthttp: starting peer 926640b1cf5e96bb...\r\n2017-05-23T07:04:13.299231490Z 2017-05-23 07:04:13.299176 I | rafthttp: started HTTP pipelining with peer 926640b1cf5e96bb\r\n2017-05-23T07:04:13.321751452Z 2017-05-23 07:04:13.321646 I | rafthttp: started streaming with peer 926640b1cf5e96bb (writer)\r\n2017-05-23T07:04:13.328276703Z 2017-05-23 07:04:13.328139 I | rafthttp: started streaming with peer 926640b1cf5e96bb (writer)\r\n2017-05-23T07:04:13.355077719Z 2017-05-23 07:04:13.354968 I | rafthttp: started peer 926640b1cf5e96bb\r\n2017-05-23T07:04:13.357459564Z 2017-05-23 07:04:13.357398 I | rafthttp: added peer 926640b1cf5e96bb\r\n2017-05-23T07:04:13.359537658Z 2017-05-23 07:04:13.359492 I | rafthttp: started streaming with peer 926640b1cf5e96bb (stream MsgApp v2 reader)\r\n2017-05-23T07:04:13.362529597Z 2017-05-23 07:04:13.362463 I | rafthttp: started streaming with peer 926640b1cf5e96bb (stream Message reader)\r\n2017-05-23T07:04:17.004583677Z 2017-05-23 07:04:17.004486 W | etcdserver: failed to reach the peerURL(http://etcd-cluster-0020.etcd-cluster.pump.svc.cluster.local:2380) of member 926640b1cf5e96bb (Get http://etcd-cluster-0020.etcd-cluster.pump.svc.cluster.local:2380/version: dial tcp 10.4.7.202:2380: getsockopt: connection refused)\r\n2017-05-23T07:04:17.004839636Z 2017-05-23 07:04:17.004791 W | etcdserver: cannot get the version of member 926640b1cf5e96bb (Get http://etcd-cluster-0020.etcd-cluster.pump.svc.cluster.local:2380/version: dial tcp 10.4.7.202:2380: getsockopt: connection refused)\r\n2017-05-23T07:04:18.365328777Z 2017-05-23 07:04:18.365234 W | rafthttp: health check for peer 926640b1cf5e96bb could not connect: dial tcp 10.4.7.202:2380: getsockopt: connection refused\r\n2017-05-23T07:04:21.395799555Z 2017-05-23 07:04:21.395706 W | etcdserver: failed to reach the peerURL(http://etcd-cluster-0020.etcd-cluster.pump.svc.cluster.local:2380) of member 926640b1cf5e96bb (Get http://etcd-cluster-0020.etcd-cluster.pump.svc.cluster.local:2380/version: read tcp 10.4.5.177:41688->10.4.7.202:2380: read: connection reset by peer)\r\n2017-05-23T07:04:21.397312909Z 2017-05-23 07:04:21.397249 W | etcdserver: cannot get the version of member 926640b1cf5e96bb (Get http://etcd-cluster-0020.etcd-cluster.pump.svc.cluster.local:2380/version: read tcp 10.4.5.177:41688->10.4.7.202:2380: read: connection reset by peer)\r\n2017-05-23T07:04:23.367652372Z 2017-05-23 07:04:23.367502 W | rafthttp: health check for peer 926640b1cf5e96bb could not connect: read tcp 10.4.5.177:41698->10.4.7.202:2380: read: connection reset by peer\r\n2017-05-23T07:04:26.603804223Z 2017-05-23 07:04:26.603660 W | etcdserver: failed to reach the peerURL(http://etcd-cluster-0020.etcd-cluster.pump.svc.cluster.local:2380) of member 926640b1cf5e96bb (Get http://etcd-cluster-0020.etcd-cluster.pump.svc.cluster.local:2380/version: dial tcp 10.4.7.202:2380: i/o timeout)\r\n2017-05-23T07:04:26.605286427Z 2017-05-23 07:04:26.605223 W | etcdserver: cannot get the version of member 926640b1cf5e96bb (Get http://etcd-cluster-0020.etcd-cluster.pump.svc.cluster.local:2380/version: dial tcp 10.4.7.202:2380: i/o timeout)\r\n2017-05-23T07:04:28.385577222Z 2017-05-23 07:04:28.385458 W | rafthttp: health check for peer 926640b1cf5e96bb could not connect: dial tcp 10.4.7.202:2380: i/o timeout\r\n2017-05-23T07:04:29.810817406Z 2017-05-23 07:04:29.810697 I | etcdserver/membership: removed member 926640b1cf5e96bb from cluster af539b1f41ca4937\r\n2017-05-23T07:04:29.810932234Z 2017-05-23 07:04:29.810909 I | rafthttp: stopping peer 926640b1cf5e96bb...\r\n2017-05-23T07:04:29.810995769Z 2017-05-23 07:04:29.810975 I | rafthttp: stopped streaming with peer 926640b1cf5e96bb (writer)\r\n2017-05-23T07:04:29.811046460Z 2017-05-23 07:04:29.811016 I | rafthttp: stopped streaming with peer 926640b1cf5e96bb (writer)\r\n2017-05-23T07:04:29.828659397Z 2017-05-23 07:04:29.828551 I | rafthttp: stopped HTTP pipelining with peer 926640b1cf5e96bb\r\n2017-05-23T07:04:29.828773018Z 2017-05-23 07:04:29.828750 I | rafthttp: stopped streaming with peer 926640b1cf5e96bb (stream MsgApp v2 reader)\r\n2017-05-23T07:04:29.828861191Z 2017-05-23 07:04:29.828833 I | rafthttp: stopped streaming with peer 926640b1cf5e96bb (stream Message reader)\r\n2017-05-23T07:04:29.828934037Z 2017-05-23 07:04:29.828915 I | rafthttp: stopped peer 926640b1cf5e96bb\r\n2017-05-23T07:04:29.828967474Z 2017-05-23 07:04:29.828952 I | rafthttp: removed peer 926640b1cf5e96bb\r\n```\r\n\r\n\r\n007 Logs:\r\n\r\n\r\n```\r\n\r\n2017-05-23T07:03:16.564593615Z 2017-05-23 07:03:16.564556 I | rafthttp: stopped peer a0bdf3ea9c1d81cc\r\n2017-05-23T07:03:16.564724790Z 2017-05-23 07:03:16.564651 I | rafthttp: removed peer a0bdf3ea9c1d81cc\r\n2017-05-23T07:03:24.721567321Z 2017-05-23 07:03:24.721430 I | etcdserver/membership: added member 5ca172185c202000 [http://etcd-cluster-0018.etcd-cluster.pump.svc.cluster.local:2380] to cluster af539b1f41ca4937\r\n2017-05-23T07:03:24.721859501Z 2017-05-23 07:03:24.721820 I | rafthttp: starting peer 5ca172185c202000...\r\n2017-05-23T07:03:24.722013354Z 2017-05-23 07:03:24.721961 I | rafthttp: started HTTP pipelining with peer 5ca172185c202000\r\n2017-05-23T07:03:24.725808923Z 2017-05-23 07:03:24.725776 I | rafthttp: started streaming with peer 5ca172185c202000 (writer)\r\n2017-05-23T07:03:24.725975363Z 2017-05-23 07:03:24.725949 I | rafthttp: started streaming with peer 5ca172185c202000 (writer)\r\n2017-05-23T07:03:24.728082571Z 2017-05-23 07:03:24.728044 I | rafthttp: started peer 5ca172185c202000\r\n2017-05-23T07:03:24.728196078Z 2017-05-23 07:03:24.728164 I | rafthttp: added peer 5ca172185c202000\r\n2017-05-23T07:03:24.728267944Z 2017-05-23 07:03:24.728245 I | rafthttp: started streaming with peer 5ca172185c202000 (stream MsgApp v2 reader)\r\n2017-05-23T07:03:24.728386304Z 2017-05-23 07:03:24.728363 I | rafthttp: started streaming with peer 5ca172185c202000 (stream Message reader)\r\n2017-05-23T07:03:29.728606458Z 2017-05-23 07:03:29.728473 W | rafthttp: health check for peer 5ca172185c202000 could not connect: dial tcp 10.4.7.201:2380: getsockopt: connection refused\r\n2017-05-23T07:03:34.731036362Z 2017-05-23 07:03:34.730883 W | rafthttp: health check for peer 5ca172185c202000 could not connect: read tcp 10.4.10.135:33896->10.4.7.201:2380: read: connection reset by peer\r\n2017-05-23T07:03:39.731425637Z 2017-05-23 07:03:39.731300 W | rafthttp: health check for peer 5ca172185c202000 could not connect: dial tcp: lookup etcd-cluster-0018.etcd-cluster.pump.svc.cluster.local on 10.7.240.10:53: no such host\r\n2017-05-23T07:03:40.868468460Z 2017-05-23 07:03:40.868357 I | etcdserver/membership: removed member 5ca172185c202000 from cluster af539b1f41ca4937\r\n2017-05-23T07:03:40.868642577Z 2017-05-23 07:03:40.868604 I | rafthttp: stopping peer 5ca172185c202000...\r\n2017-05-23T07:03:40.868711363Z 2017-05-23 07:03:40.868677 I | rafthttp: stopped streaming with peer 5ca172185c202000 (writer)\r\n2017-05-23T07:03:40.868798792Z 2017-05-23 07:03:40.868754 I | rafthttp: stopped streaming with peer 5ca172185c202000 (writer)\r\n2017-05-23T07:03:40.868898434Z 2017-05-23 07:03:40.868877 I | rafthttp: stopped HTTP pipelining with peer 5ca172185c202000\r\n2017-05-23T07:03:40.868983302Z 2017-05-23 07:03:40.868963 I | rafthttp: stopped streaming with peer 5ca172185c202000 (stream MsgApp v2 reader)\r\n2017-05-23T07:03:40.869043652Z 2017-05-23 07:03:40.869008 I | rafthttp: stopped streaming with peer 5ca172185c202000 (stream Message reader)\r\n2017-05-23T07:03:40.869080192Z 2017-05-23 07:03:40.869061 I | rafthttp: stopped peer 5ca172185c202000\r\n2017-05-23T07:03:40.869150891Z 2017-05-23 07:03:40.869131 I | rafthttp: removed peer 5ca172185c202000\r\n2017-05-23T07:03:49.001384545Z 2017-05-23 07:03:49.001243 I | etcdserver/membership: added member d7cea5d10f262f59 [http://etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local:2380] to cluster af539b1f41ca4937\r\n2017-05-23T07:03:49.001577791Z 2017-05-23 07:03:49.001474 I | rafthttp: starting peer d7cea5d10f262f59...\r\n2017-05-23T07:03:49.001650959Z 2017-05-23 07:03:49.001591 I | rafthttp: started HTTP pipelining with peer d7cea5d10f262f59\r\n2017-05-23T07:03:49.011304638Z 2017-05-23 07:03:49.011206 I | rafthttp: started streaming with peer d7cea5d10f262f59 (writer)\r\n2017-05-23T07:03:49.012117470Z 2017-05-23 07:03:49.012058 I | rafthttp: started peer d7cea5d10f262f59\r\n2017-05-23T07:03:49.012206851Z 2017-05-23 07:03:49.012151 I | rafthttp: added peer d7cea5d10f262f59\r\n2017-05-23T07:03:49.012281019Z 2017-05-23 07:03:49.012235 I | rafthttp: started streaming with peer d7cea5d10f262f59 (stream MsgApp v2 reader)\r\n2017-05-23T07:03:49.012456996Z 2017-05-23 07:03:49.012425 I | rafthttp: started streaming with peer d7cea5d10f262f59 (writer)\r\n2017-05-23T07:03:49.012503020Z 2017-05-23 07:03:49.012476 I | rafthttp: started streaming with peer d7cea5d10f262f59 (stream Message reader)\r\n2017-05-23T07:03:54.012487367Z 2017-05-23 07:03:54.012338 W | rafthttp: health check for peer d7cea5d10f262f59 could not connect: dial tcp 10.4.9.45:2380: getsockopt: connection refused\r\n2017-05-23T07:03:57.097301882Z 2017-05-23 07:03:57.097200 W | etcdserver/membership: Reject add member request: the number of started member (2) will be less than the quorum number of the cluster (3)\r\n2017-05-23T07:03:57.097557412Z 2017-05-23 07:03:57.097503 W | etcdserver: not enough started members, rejecting member add {ID:98ea03a7831ff29e RaftAttributes:{PeerURLs:[http://etcd-cluster-0020.etcd-cluster.pump.svc.cluster.local:2380]} Attributes:{Name: ClientURLs:[]}}\r\n2017-05-23T07:03:59.012857577Z 2017-05-23 07:03:59.012734 W | rafthttp: health check for peer d7cea5d10f262f59 could not connect: dial tcp 10.4.9.45:2380: getsockopt: connection refused\r\n2017-05-23T07:04:04.013108170Z 2017-05-23 07:04:04.013013 W | rafthttp: health check for peer d7cea5d10f262f59 could not connect: dial tcp: lookup etcd-cluster-0019.etcd-cluster.pump.svc.cluster.local on 10.7.240.10:53: no such host\r\n2017-05-23T07:04:05.208384181Z 2017-05-23 07:04:05.208234 I | etcdserver/membership: removed member d7cea5d10f262f59 from cluster af539b1f41ca4937\r\n2017-05-23T07:04:05.208497178Z 2017-05-23 07:04:05.208433 I | rafthttp: stopping peer d7cea5d10f262f59...\r\n2017-05-23T07:04:05.208559688Z 2017-05-23 07:04:05.208480 I | rafthttp: stopped streaming with peer d7cea5d10f262f59 (writer)\r\n2017-05-23T07:04:05.208696147Z 2017-05-23 07:04:05.208620 I | rafthttp: stopped streaming with peer d7cea5d10f262f59 (writer)\r\n2017-05-23T07:04:05.208716127Z 2017-05-23 07:04:05.208695 I | rafthttp: stopped HTTP pipelining with peer d7cea5d10f262f59\r\n2017-05-23T07:04:05.208785717Z 2017-05-23 07:04:05.208719 I | rafthttp: stopped streaming with peer d7cea5d10f262f59 (stream MsgApp v2 reader)\r\n2017-05-23T07:04:05.208802024Z 2017-05-23 07:04:05.208783 I | rafthttp: stopped streaming with peer d7cea5d10f262f59 (stream Message reader)\r\n2017-05-23T07:04:05.208854770Z 2017-05-23 07:04:05.208806 I | rafthttp: stopped peer d7cea5d10f262f59\r\n2017-05-23T07:04:05.208883118Z 2017-05-23 07:04:05.208854 I | rafthttp: removed peer d7cea5d10f262f59\r\n2017-05-23T07:04:13.325867208Z 2017-05-23 07:04:13.325730 I | etcdserver/membership: added member 926640b1cf5e96bb [http://etcd-cluster-0020.etcd-cluster.pump.svc.cluster.local:2380] to cluster af539b1f41ca4937\r\n2017-05-23T07:04:13.325999385Z 2017-05-23 07:04:13.325966 I | rafthttp: starting peer 926640b1cf5e96bb...\r\n2017-05-23T07:04:13.326071011Z 2017-05-23 07:04:13.326048 I | rafthttp: started HTTP pipelining with peer 926640b1cf5e96bb\r\n2017-05-23T07:04:13.327882972Z 2017-05-23 07:04:13.327830 I | rafthttp: started peer 926640b1cf5e96bb\r\n2017-05-23T07:04:13.327945595Z 2017-05-23 07:04:13.327916 I | rafthttp: added peer 926640b1cf5e96bb\r\n2017-05-23T07:04:13.328161236Z 2017-05-23 07:04:13.328118 I | rafthttp: started streaming with peer 926640b1cf5e96bb (writer)\r\n2017-05-23T07:04:13.328212702Z 2017-05-23 07:04:13.328192 I | rafthttp: started streaming with peer 926640b1cf5e96bb (writer)\r\n2017-05-23T07:04:13.328296332Z 2017-05-23 07:04:13.328274 I | rafthttp: started streaming with peer 926640b1cf5e96bb (stream MsgApp v2 reader)\r\n\r\n```\r\nOperator Config:\r\n\r\n```\r\napiVersion: extensions/v1beta1\r\nkind: Deployment\r\nmetadata:\r\n  name: etcd-operator\r\n  namespace: pump\r\nspec:\r\n  replicas: 1\r\n  template:\r\n    metadata:\r\n      labels:\r\n        name: etcd-operator\r\n    spec:\r\n      containers:\r\n      - name: etcd-operator\r\n        image: quay.io/coreos/etcd-operator:v0.3.0\r\n        resources:\r\n              limits:\r\n                cpu: 125m\r\n                memory: 200Mi\r\n              requests:\r\n                cpu: 125m\r\n                memory: 100Mi\r\n        env:\r\n        - name: MY_POD_NAMESPACE\r\n          valueFrom:\r\n            fieldRef:\r\n              fieldPath: metadata.namespace\r\n        - name: MY_POD_NAME\r\n          valueFrom:\r\n            fieldRef:\r\n              fieldPath: metadata.name\r\n\r\n```\r\nCluster Config:\r\n\r\n```\r\napiVersion: \"etcd.coreos.com/v1beta1\"\r\nkind: \"Cluster\"\r\nmetadata:\r\n  name: \"etcd-cluster\"\r\nspec:\r\n  size: 5\r\n  version: \"3.1.8\"\r\n  pod:\r\n    antiAffinity: true\r\n  backup:\r\n    backupIntervalInSecond: 600\r\n    maxBackups: 24\r\n    storageType: \"PersistentVolume\"\r\n    pv:\r\n      volumeSizeInMB: 512\r\n#  restore:\r\n#    backupClusterName: \"etcd-cluster\"\r\n#    storageType: \"PersistentVolume\"\r\n```\r\n![screen shot 2017-05-23 at 3 06 26 pm](https://cloud.githubusercontent.com/assets/1250044/26342167/884a80e2-3fc9-11e7-992c-bee3d27aa472.png)\r\n"},"comment":{"url":"https://api.github.com/repos/coreos/etcd-operator/issues/comments/303551154","html_url":"https://github.com/coreos/etcd-operator/issues/1121#issuecomment-303551154","issue_url":"https://api.github.com/repos/coreos/etcd-operator/issues/1121","id":303551154,"user":{"login":"xiang90","id":4479947,"avatar_url":"https://avatars3.githubusercontent.com/u/4479947?v=3","gravatar_id":"","url":"https://api.github.com/users/xiang90","html_url":"https://github.com/xiang90","followers_url":"https://api.github.com/users/xiang90/followers","following_url":"https://api.github.com/users/xiang90/following{/other_user}","gists_url":"https://api.github.com/users/xiang90/gists{/gist_id}","starred_url":"https://api.github.com/users/xiang90/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiang90/subscriptions","organizations_url":"https://api.github.com/users/xiang90/orgs","repos_url":"https://api.github.com/users/xiang90/repos","events_url":"https://api.github.com/users/xiang90/events{/privacy}","received_events_url":"https://api.github.com/users/xiang90/received_events","type":"User","site_admin":false},"created_at":"2017-05-23T22:30:04Z","updated_at":"2017-05-23T22:30:04Z","body":"@ravindranathakila Can you try to reproduce it? "}}
{"actor":{"display_login":"xiang90"}}