{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd/issues/8899","repository_url":"https://api.github.com/repos/coreos/etcd","labels_url":"https://api.github.com/repos/coreos/etcd/issues/8899/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd/issues/8899/comments","events_url":"https://api.github.com/repos/coreos/etcd/issues/8899/events","html_url":"https://github.com/coreos/etcd/issues/8899","id":275585915,"number":8899,"title":"Watchers not getting updates","user":{"login":"cw9","id":6710599,"avatar_url":"https://avatars0.githubusercontent.com/u/6710599?v=4","gravatar_id":"","url":"https://api.github.com/users/cw9","html_url":"https://github.com/cw9","followers_url":"https://api.github.com/users/cw9/followers","following_url":"https://api.github.com/users/cw9/following{/other_user}","gists_url":"https://api.github.com/users/cw9/gists{/gist_id}","starred_url":"https://api.github.com/users/cw9/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cw9/subscriptions","organizations_url":"https://api.github.com/users/cw9/orgs","repos_url":"https://api.github.com/users/cw9/repos","events_url":"https://api.github.com/users/cw9/events{/privacy}","received_events_url":"https://api.github.com/users/cw9/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2017-11-21T04:59:31Z","updated_at":"2017-11-25T21:41:55Z","closed_at":null,"author_association":"NONE","body":"etcd Version: 3.2.4\r\nGit SHA: c31bec0\r\nGo Version: go1.8.3\r\nGo OS/Arch: linux/amd64\r\nClient version: 8ba2897a21e4fc51b298ca553d251318425f93ae\r\n\r\nWe are currently running a 5 node etcd cluster with 200k+ watchers, we have been rarely writing to the cluster. Today when we updated the value of a key, less than 1% of all the watchers did not receive the update, this is not the first time we observed this issue.\r\n\r\nSome more debugging on a problematic application process shows this:\r\n```\r\ngoroutine 79 [select, 4361 minutes]:\r\ninternal_code/vendor/github.com/coreos/etcd/clientv3.(*watchGrpcStream).serveSubstream(0xc4203615e0, 0xc420aa0000, 0xc42055ec00)\r\n    /internal_code/vendor/github.com/coreos/etcd/clientv3/watch.go:591 +0x526\r\ncreated by internal_code/vendor/github.com/coreos/etcd/clientv3.(*watchGrpcStream).run\r\n    /internal_code/vendor/github.com/coreos/etcd/clientv3/watch.go:434 +0x47b\r\n```\r\nand we found this in the etcd log:\r\n`Nov 18 01:54:51 {etcd_hostname} etcd[53042]: transport: http2Server.HandleStreams failed to read frame: read tcp {etcd_ip}:2379->{application_ip}:44460: read: connection timed out`\r\n\r\nNot sure if this is red herring but the time of the error event does kinda match the waiting time on the goroutine. We do see this waiting time on a well-behaved box to be a few minutes and we think it's because we had `WithProgressNotify` option enabled by default, can you verify this theory?\r\n\r\nWe also see `read: connection timed out` and `read: connection reset by peer` pretty often in the logs.\r\n\r\nWe are not able to reproduce this issue, but restarting the problematic application could fix the issue. There are still a few applications running in the problematic state, they all seem to have connections to every etcd box, just won't be able to receive any updates. Is there any thing we could try to debug this?\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/coreos/etcd/issues/comments/346968073","html_url":"https://github.com/coreos/etcd/issues/8899#issuecomment-346968073","issue_url":"https://api.github.com/repos/coreos/etcd/issues/8899","id":346968073,"user":{"login":"xiang90","id":4479947,"avatar_url":"https://avatars0.githubusercontent.com/u/4479947?v=4","gravatar_id":"","url":"https://api.github.com/users/xiang90","html_url":"https://github.com/xiang90","followers_url":"https://api.github.com/users/xiang90/followers","following_url":"https://api.github.com/users/xiang90/following{/other_user}","gists_url":"https://api.github.com/users/xiang90/gists{/gist_id}","starred_url":"https://api.github.com/users/xiang90/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xiang90/subscriptions","organizations_url":"https://api.github.com/users/xiang90/orgs","repos_url":"https://api.github.com/users/xiang90/repos","events_url":"https://api.github.com/users/xiang90/events{/privacy}","received_events_url":"https://api.github.com/users/xiang90/received_events","type":"User","site_admin":false},"created_at":"2017-11-25T21:41:55Z","updated_at":"2017-11-25T21:41:55Z","author_association":"CONTRIBUTOR","body":"@xichen2020 \r\n\r\nPlease enable keepalive and give it another try."}}
{"actor":{"display_login":"xiang90"}}