{"action":"opened","issue":{"url":"https://api.github.com/repos/rancher/rancher/issues/9891","repository_url":"https://api.github.com/repos/rancher/rancher","labels_url":"https://api.github.com/repos/rancher/rancher/issues/9891/labels{/name}","comments_url":"https://api.github.com/repos/rancher/rancher/issues/9891/comments","events_url":"https://api.github.com/repos/rancher/rancher/issues/9891/events","html_url":"https://github.com/rancher/rancher/issues/9891","id":258108491,"number":9891,"title":"etcd(v3) fails to join the cluster on a 4 host setup with one host down","user":{"login":"soumyalj","id":18536626,"avatar_url":"https://avatars1.githubusercontent.com/u/18536626?v=4","gravatar_id":"","url":"https://api.github.com/users/soumyalj","html_url":"https://github.com/soumyalj","followers_url":"https://api.github.com/users/soumyalj/followers","following_url":"https://api.github.com/users/soumyalj/following{/other_user}","gists_url":"https://api.github.com/users/soumyalj/gists{/gist_id}","starred_url":"https://api.github.com/users/soumyalj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumyalj/subscriptions","organizations_url":"https://api.github.com/users/soumyalj/orgs","repos_url":"https://api.github.com/users/soumyalj/repos","events_url":"https://api.github.com/users/soumyalj/events{/privacy}","received_events_url":"https://api.github.com/users/soumyalj/received_events","type":"User","site_admin":false},"labels":[{"id":145695744,"url":"https://api.github.com/repos/rancher/rancher/labels/kind/bug","name":"kind/bug","color":"fc2929","default":false}],"state":"open","locked":false,"assignee":{"login":"galal-hussein","id":3309968,"avatar_url":"https://avatars1.githubusercontent.com/u/3309968?v=4","gravatar_id":"","url":"https://api.github.com/users/galal-hussein","html_url":"https://github.com/galal-hussein","followers_url":"https://api.github.com/users/galal-hussein/followers","following_url":"https://api.github.com/users/galal-hussein/following{/other_user}","gists_url":"https://api.github.com/users/galal-hussein/gists{/gist_id}","starred_url":"https://api.github.com/users/galal-hussein/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/galal-hussein/subscriptions","organizations_url":"https://api.github.com/users/galal-hussein/orgs","repos_url":"https://api.github.com/users/galal-hussein/repos","events_url":"https://api.github.com/users/galal-hussein/events{/privacy}","received_events_url":"https://api.github.com/users/galal-hussein/received_events","type":"User","site_admin":false},"assignees":[{"login":"galal-hussein","id":3309968,"avatar_url":"https://avatars1.githubusercontent.com/u/3309968?v=4","gravatar_id":"","url":"https://api.github.com/users/galal-hussein","html_url":"https://github.com/galal-hussein","followers_url":"https://api.github.com/users/galal-hussein/followers","following_url":"https://api.github.com/users/galal-hussein/following{/other_user}","gists_url":"https://api.github.com/users/galal-hussein/gists{/gist_id}","starred_url":"https://api.github.com/users/galal-hussein/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/galal-hussein/subscriptions","organizations_url":"https://api.github.com/users/galal-hussein/orgs","repos_url":"https://api.github.com/users/galal-hussein/repos","events_url":"https://api.github.com/users/galal-hussein/events{/privacy}","received_events_url":"https://api.github.com/users/galal-hussein/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/rancher/rancher/milestones/105","html_url":"https://github.com/rancher/rancher/milestone/105","labels_url":"https://api.github.com/repos/rancher/rancher/milestones/105/labels","id":2689386,"number":105,"title":"September 2017","description":"","creator":{"login":"deniseschannon","id":11410997,"avatar_url":"https://avatars1.githubusercontent.com/u/11410997?v=4","gravatar_id":"","url":"https://api.github.com/users/deniseschannon","html_url":"https://github.com/deniseschannon","followers_url":"https://api.github.com/users/deniseschannon/followers","following_url":"https://api.github.com/users/deniseschannon/following{/other_user}","gists_url":"https://api.github.com/users/deniseschannon/gists{/gist_id}","starred_url":"https://api.github.com/users/deniseschannon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deniseschannon/subscriptions","organizations_url":"https://api.github.com/users/deniseschannon/orgs","repos_url":"https://api.github.com/users/deniseschannon/repos","events_url":"https://api.github.com/users/deniseschannon/events{/privacy}","received_events_url":"https://api.github.com/users/deniseschannon/received_events","type":"User","site_admin":false},"open_issues":55,"closed_issues":20,"state":"open","created_at":"2017-08-08T21:56:46Z","updated_at":"2017-09-15T17:20:07Z","due_on":"2017-09-30T07:00:00Z","closed_at":null},"comments":0,"created_at":"2017-09-15T17:20:07Z","updated_at":"2017-09-15T17:20:07Z","closed_at":null,"author_association":"NONE","body":"Rancher versions:\r\nrancher/server: master\r\nkubernetes (if applicable):v1.7.2\r\n\r\n**Docker version: (docker version,docker info preferred)**1.12.6\r\n\r\n**Operating system and kernel: (cat /etc/os-release, uname -r preferred)**Ubuntu\r\n\r\n**Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)**DO\r\n\r\n**Setup details: (single node rancher vs. HA rancher, internal DB vs. external DB)**Single node\r\n\r\n**Environment Template: (Cattle/Kubernetes/Swarm/Mesos)**Kubernetes\r\n\r\nSteps to Reproduce:\r\n\r\n1. Create a 4 node setup and 3 etcd nodes land up as below:\r\nh1 - etcd_1\r\nh2 - etcd_2\r\nh3 - etcd_3\r\nh4 - None\r\n2. power off h1. etcd_1 lands on h4\r\n3. power on h1 and then power off h4. This will result in ectd_1 starting with stale volume  on h1.\r\netcd_1 fails to join the cluster:\r\netcdctl cluster-health from etcd_1\r\n```\r\nbash-4.3# etcdctl cluster-health\r\ncluster may be unhealthy: failed to list members\r\nError:  client: etcd cluster is unavailable or misconfigured\r\nerror #0: client: endpoint http://127.0.0.1:2379 exceeded header timeout\r\nerror #1: dial tcp 127.0.0.1:4001: getsockopt: connection refused\r\n```\r\nAlthough, member list from the other healthy etcd lists all the 3 members\r\n```\r\nbash-4.3# etcdctl member list\r\n48b66f9a46bbbcc7: name=etcd-3 peerURLs=http://10.42.196.10:2380 clientURLs=http:\r\n//10.42.196.10:2379 isLeader=true\r\n8caf75890e14a4a4: name=etcd-1 peerURLs=http://10.42.112.148:2380 clientURLs=http\r\n://10.42.112.148:2379 isLeader=false\r\ncd4017ba403e4af1: name=etcd-2 peerURLs=http://10.42.30.197:2380 clientURLs=http:\r\n//10.42.30.197:2379 isLeader=false\r\n\r\n```\r\nLogs of etcd_1 below:\r\n```\r\n9/15/2017 10:11:05 AM2017-09-15 17:11:05.998409 E | rafthttp: failed to find member 48b66f9a46bbbcc7 in cluster 72d1e88a0ae55537\r\n9/15/2017 10:11:06 AM2017-09-15 17:11:06.010735 E | rafthttp: failed to find member 48b66f9a46bbbcc7 in cluster 72d1e88a0ae55537\r\n9/15/2017 10:11:06 AM2017-09-15 17:11:06.027104 E | rafthttp: failed to find member 8caf75890e14a4a4 in cluster 72d1e88a0ae55537\r\n9/15/2017 10:11:06 AM2017-09-15 17:11:06.028892 E | rafthttp: failed to find member 8caf75890e14a4a4 in cluster 72d1e88a0ae55537\r\n9/15/2017 10:11:06 AM2017-09-15 17:11:06.107075 E | rafthttp: failed to find member 48b66f9a46bbbcc7 in cluster 72d1e88a0ae55537\r\n9/15/2017 10:11:06 AM2017-09-15 17:11:06.114464 E | rafthttp: failed to find member \r\n```\r\n"}}
{"actor":{"display_login":"soumyalj"}}