{"action":"opened","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/46964","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/46964/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/46964/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/46964/events","html_url":"https://github.com/kubernetes/kubernetes/issues/46964","id":233590755,"number":46964,"title":"apiserver etcd v3 client does not close connections to dead etcd node","user":{"login":"macpoint","id":2634756,"avatar_url":"https://avatars0.githubusercontent.com/u/2634756?v=3","gravatar_id":"","url":"https://api.github.com/users/macpoint","html_url":"https://github.com/macpoint","followers_url":"https://api.github.com/users/macpoint/followers","following_url":"https://api.github.com/users/macpoint/following{/other_user}","gists_url":"https://api.github.com/users/macpoint/gists{/gist_id}","starred_url":"https://api.github.com/users/macpoint/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/macpoint/subscriptions","organizations_url":"https://api.github.com/users/macpoint/orgs","repos_url":"https://api.github.com/users/macpoint/repos","events_url":"https://api.github.com/users/macpoint/events{/privacy}","received_events_url":"https://api.github.com/users/macpoint/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-06-05T13:39:49Z","updated_at":"2017-06-05T13:39:49Z","closed_at":null,"body":"Hi,\r\n\r\nrunning 1.6.4 on etcd 3.0.17 in the following setup:\r\n\r\n- 3 etcd nodes (bootstrapped according to the [clustering guide](https://coreos.com/etcd/docs/latest/op-guide/clustering.html))\r\n- 9 kubernetes cluster nodes all of them running kube-apiserver (--apiserver-count=9) with --bind-address to local kube-apiserver instance. All instances pointing to the same etcd cluster (--etcd-servers=http://IP1:4001,http://IP2:4001,http://IP3:4001)\r\n\r\nI tried to simulate 1 etcd node crash by simply turing off one of the machines. Result:\r\n- timeouts running kubectl (get nodes, get pods) commands\r\n- lots of these error messages in kube-apiserver logs: \"status.go:62] apiserver received an error that is not an metav1.Status: http: Handler timeout\"\r\n- ETCD correctly reports unhealthy node, but healthy cluster\r\n\r\nInspecting netstat shows lots of ESTABLISHED connections to the dead etcd server, port 4001 - these connections are cleared only by restarting the kube-apiserver(s).\r\n\r\nWorkarounds:\r\n- Switching back to etcd 2.2.1 and using --storage-backend=etcd2 ... ESTABLISHED connections are cleared within several seconds\r\n- Leaving etcd 3.0.17 and using --storage-backend=etcd2 ... ESTABLISHED connections are cleared within several seconds\r\n\r\nIn both above cases kubectl shows expected results. The dead node is evicted (NotReady), other nodes & pods are in tact.\r\n\r\nI noticed that [etcd v2](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/storagebackend/factory/etcd2.go) client contains TCP KeepAlive timeout, while [etcd v3](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/pkg/storage/storagebackend/factory/etcd3.go) client does not. Could this be related to the described scenario?\r\n\r\n<!--\r\nIf this is a BUG REPORT, please:\r\n  - Fill in as much of the template below as you can.  If you leave out\r\n    information, we can't help you as well.\r\n\r\nIf this is a FEATURE REQUEST, please:\r\n  - Describe *in detail* the feature/behavior/change you'd like to see.\r\n\r\nIn both cases, be ready for followup questions, and please respond in a timely\r\nmanner.  If we can't reproduce a bug or think a feature already exists, we\r\nmight close your issue.  If we're wrong, PLEASE feel free to reopen it and\r\nexplain why.\r\n-->\r\n\r\n**Kubernetes version** \r\n`Client Version: version.Info{Major:\"1\", Minor:\"6\", GitVersion:\"v1.6.4\", GitCommit:\"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\", GitTreeState:\"clean\", BuildDate:\"2017-05-19T18:44:27Z\", GoVersion:\"go1.7.5\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"6\", GitVersion:\"v1.6.4\", GitCommit:\"d6f433224538d4f9ca2f7ae19b252e6fcb66a3ae\", GitTreeState:\"clean\", BuildDate:\"2017-05-19T18:33:17Z\", GoVersion:\"go1.7.5\", Compiler:\"gc\", Platform:\"linux/amd64\"}`\r\n\r\n\r\n**Environment**:\r\n- **OS**:\r\nNAME=\"Oracle Linux Server\"\r\nVERSION=\"6.7\"\r\nID=\"ol\"\r\nVERSION_ID=\"6.7\"\r\nPRETTY_NAME=\"Oracle Linux Server 6.7\"\r\nANSI_COLOR=\"0;31\"\r\nCPE_NAME=\"cpe:/o:oracle:linux:6:7\"\r\nHOME_URL=\"https://linux.oracle.com/\"\r\nBUG_REPORT_URL=\"https://bugzilla.oracle.com/\"\r\nORACLE_BUGZILLA_PRODUCT=\"Oracle Linux 6\"\r\nORACLE_BUGZILLA_PRODUCT_VERSION=6.7\r\nORACLE_SUPPORT_PRODUCT=\"Oracle Linux\"\r\nORACLE_SUPPORT_PRODUCT_VERSION=6.7\r\n- **Kernel** \r\n`Linux ngi-test-mgmt 4.1.12-94.3.4.el6uek.x86_64 #2 SMP Mon May 15 14:10:25 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux`\r\n\r\n"}}
{"actor":{"display_login":"macpoint"}}