{"action":"closed","issue":{"url":"https://api.github.com/repos/weaveworks/weave/issues/3021","repository_url":"https://api.github.com/repos/weaveworks/weave","labels_url":"https://api.github.com/repos/weaveworks/weave/issues/3021/labels{/name}","comments_url":"https://api.github.com/repos/weaveworks/weave/issues/3021/comments","events_url":"https://api.github.com/repos/weaveworks/weave/issues/3021/events","html_url":"https://github.com/weaveworks/weave/issues/3021","id":236723677,"number":3021,"title":"weave attempting to connect to deleted nodes, and failing with functioning ones","user":{"login":"deitch","id":282526,"avatar_url":"https://avatars3.githubusercontent.com/u/282526?v=4","gravatar_id":"","url":"https://api.github.com/users/deitch","html_url":"https://github.com/deitch","followers_url":"https://api.github.com/users/deitch/followers","following_url":"https://api.github.com/users/deitch/following{/other_user}","gists_url":"https://api.github.com/users/deitch/gists{/gist_id}","starred_url":"https://api.github.com/users/deitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/deitch/subscriptions","organizations_url":"https://api.github.com/users/deitch/orgs","repos_url":"https://api.github.com/users/deitch/repos","events_url":"https://api.github.com/users/deitch/events{/privacy}","received_events_url":"https://api.github.com/users/deitch/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2017-06-18T14:16:18Z","updated_at":"2017-12-04T15:31:13Z","closed_at":"2017-12-04T15:31:12Z","author_association":"CONTRIBUTOR","body":"<!--\r\nHi, thank you for opening an issue!\r\nBefore hitting the button...\r\n\r\n** Is this a REQUEST FOR HELP? **\r\nIf so, please have a look at:\r\n- our FAQ: https://www.weave.works/docs/net/latest/faq/\r\n- our troubleshooting page: https://www.weave.works/docs/net/latest/troubleshooting/\r\n- our help page, to choose the best channel (Slack, etc.) to reach out: https://www.weave.works/help/\r\n\r\n** Is this a FEATURE REQUEST? **\r\nIf so, please search existing feature requests, and if you find a similar one, up-vote it and/or add your comments to it instead.\r\nIf you did not find a similar one, please describe in details:\r\n- why: your use-case, specific constraints you may have, etc.\r\n- what: the feature/behaviour/change you would like to see in Weave Net\r\nDo not hesitate, when appropriate, to share the exact commands or API you would like, and/or to share a diagram (e.g.: asciiflow.com): \"a picture is worth a thousand words\".\r\n\r\n** Is this a BUG REPORT? **\r\nPlease fill in as much of the template below as you can.\r\n\r\nThank you!\r\n-->\r\n\r\n## What you expected to happen?\r\n\r\nweave-kube should be able to connect consistently to functional nodes and ignore deleted ones\r\n\r\n## What happened?\r\n<!-- Error message, actual behaviour, etc. -->\r\n\r\nLogs show \"flapping\", in/out, but constantly tries to connect to deleted nodes\r\n\r\n## How to reproduce it?\r\n<!-- Specific steps, as minimally and precisely as possible. -->\r\n\r\nUnsure. I am a little nervous to delete the DaemonSet for fear of losing the problem until it shows up again in prod\r\n\r\n## Anything else we need to know?\r\n<!-- Cloud provider? Hardware? How did you configure your cluster? Kubernetes YAML, KOPS, etc. -->\r\nRunning on kube on AWS. \r\n\r\n## Versions:\r\n<!-- Please paste in the output of these commands; 'kubectl' only if using Kubernetes -->\r\n```\r\n$ weave version\r\n1.9.8  # image version; running `weave version` doesn't really work on kube, in weave-kube container gives \"docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?.\"\r\n$ docker version\r\nClient:\r\n Version:      1.12.3\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   34a2ead\r\n Built:\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.12.3\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   34a2ead\r\n Built:\r\n OS/Arch:      linux/amd64\r\n$ uname -a\r\nLinux ip-10-50-20-214.ec2.internal 4.7.3-coreos-r2 #1 SMP Sun Jan 8 00:32:25 UTC 2017 x86_64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz GenuineIntel GNU/Linux\r\n$ kubectl version\r\n1.6.2\r\n```\r\n\r\n## Logs:\r\n```\r\n$ docker logs weave\r\n```\r\nor, if using Kubernetes:\r\n```\r\n$ kubectl logs -n kube-system <weave-net-pod> weave\r\nINFO: 2017/06/18 13:45:59.381055 ->[10.50.21.194:54097] connection accepted\r\nINFO: 2017/06/18 13:45:59.382893 ->[10.50.21.194:54097|b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)]: connection ready; using protocol version 2\r\nINFO: 2017/06/18 13:45:59.383012 overlay_switch ->[b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)] using fastdp\r\nINFO: 2017/06/18 13:45:59.383055 ->[10.50.21.194:54097|b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)]: connection added (new peer)\r\nINFO: 2017/06/18 13:45:59.386618 ->[10.50.21.194:54097|b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)]: connection fully established\r\nINFO: 2017/06/18 13:45:59.387055 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)\r\nINFO: 2017/06/18 13:45:59.388897 sleeve ->[10.50.21.194:6783|b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)]: Effective MTU verified at 8939\r\nINFO: 2017/06/18 13:45:59.561696 Discovered remote MAC 9e:7d:e6:dd:3b:60 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:45:59.949617 Discovered remote MAC 36:37:42:ca:44:59 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:46:00.144553 Discovered remote MAC f6:58:af:e6:75:a0 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:46:00.144899 Discovered remote MAC ba:db:51:ef:5c:f5 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:46:00.164246 Discovered remote MAC 72:27:02:61:0b:c5 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:46:01.111513 Discovered remote MAC 3e:ab:04:52:eb:c8 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:46:01.565576 Discovered remote MAC 0e:db:2c:c9:f4:e6 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:46:01.736969 Discovered remote MAC 8e:e3:b0:fe:8c:db at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nERRO: 2017/06/18 13:46:02.432131 Captured frame from MAC (3e:ab:04:52:eb:c8) to (fa:d2:49:2d:12:73) associated with another peer b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nERRO: 2017/06/18 13:46:02.432232 Captured frame from MAC (3e:ab:04:52:eb:c8) to (fa:d2:49:2d:12:73) associated with another peer b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:46:02.531194 Discovered remote MAC 4a:46:51:1b:3c:ee at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:46:02.896173 Discovered remote MAC 82:93:18:bf:3b:dd at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:46:04.847660 Discovered remote MAC e6:44:7a:30:34:15 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:46:08.734704 Discovered remote MAC 72:1e:b2:82:93:c4 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:46:12.827627 Discovered remote MAC b2:b0:8e:9e:92:01 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:48:26.035363 ->[10.50.21.91:6783] attempting connection\r\nINFO: 2017/06/18 13:48:43.133397 ->[10.50.20.63:6783] attempting connection\r\nINFO: 2017/06/18 13:48:46.139066 ->[10.50.20.63:6783] error during connection attempt: dial tcp4 :0->10.50.20.63:6783: getsockopt: no route to host\r\nINFO: 2017/06/18 13:50:33.249061 ->[10.50.21.91:6783] error during connection attempt: dial tcp4 :0->10.50.21.91:6783: getsockopt: connection timed out\r\nINFO: 2017/06/18 13:52:21.554919 ->[10.50.21.226:6783] attempting connection\r\nINFO: 2017/06/18 13:54:28.769024 ->[10.50.21.226:6783] error during connection attempt: dial tcp4 :0->10.50.21.226:6783: getsockopt: connection timed out\r\nINFO: 2017/06/18 13:54:49.098034 Expired MAC a6:4c:16:6a:67:3a at c6:f8:08:48:aa:a4(ip-10-50-20-144.ec2.internal)\r\nINFO: 2017/06/18 13:54:49.098071 Expired MAC 9e:d4:27:92:01:2b at 12:4a:1a:49:54:d8(ip-10-50-20-214.ec2.internal)\r\nINFO: 2017/06/18 13:55:49.098291 Expired MAC 7a:12:89:41:90:fe at c6:f8:08:48:aa:a4(ip-10-50-20-144.ec2.internal)\r\nINFO: 2017/06/18 13:55:49.098333 Expired MAC c6:f8:08:48:aa:a4 at c6:f8:08:48:aa:a4(ip-10-50-20-144.ec2.internal)\r\nINFO: 2017/06/18 13:56:06.702540 ->[10.50.20.63:6783] attempting connection\r\nINFO: 2017/06/18 13:56:09.709158 ->[10.50.20.63:6783] error during connection attempt: dial tcp4 :0->10.50.20.63:6783: getsockopt: no route to host\r\nINFO: 2017/06/18 13:56:30.050728 ->[10.50.21.91:6783] attempting connection\r\nINFO: 2017/06/18 13:56:49.098541 Expired MAC 12:4a:1a:49:54:d8 at 12:4a:1a:49:54:d8(ip-10-50-20-214.ec2.internal)\r\nINFO: 2017/06/18 13:58:37.345030 ->[10.50.21.91:6783] error during connection attempt: dial tcp4 :0->10.50.21.91:6783: getsockopt: connection timed out\r\nINFO: 2017/06/18 13:59:18.085603 ->[10.50.20.63:6783] attempting connection\r\nINFO: 2017/06/18 13:59:21.091068 ->[10.50.20.63:6783] error during connection attempt: dial tcp4 :0->10.50.20.63:6783: getsockopt: no route to host\r\nINFO: 2017/06/18 13:59:49.099264 Expired MAC 3e:ab:04:52:eb:c8 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:59:49.099324 Expired MAC 8e:e3:b0:fe:8c:db at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:59:49.099352 Expired MAC 9e:7d:e6:dd:3b:60 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:59:49.099377 Expired MAC f6:58:af:e6:75:a0 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:59:49.099425 Expired MAC 82:93:18:bf:3b:dd at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:59:49.099438 Expired MAC ba:db:51:ef:5c:f5 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:59:49.099450 Expired MAC 4a:46:51:1b:3c:ee at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:59:49.099462 Expired MAC e6:44:7a:30:34:15 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 13:59:49.099474 Expired MAC 72:27:02:61:0b:c5 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 14:01:17.985491 Discovered local MAC ba:b6:78:61:2d:9d\r\nINFO: 2017/06/18 14:02:22.074616 ->[10.50.20.63:6783] attempting connection\r\nINFO: 2017/06/18 14:02:25.081171 ->[10.50.20.63:6783] error during connection attempt: dial tcp4 :0->10.50.20.63:6783: getsockopt: no route to host\r\nINFO: 2017/06/18 14:02:55.986007 ->[10.50.21.226:6783] attempting connection\r\nINFO: 2017/06/18 14:04:50.546119 ->[10.50.21.91:6783] attempting connection\r\nINFO: 2017/06/18 14:05:03.265054 ->[10.50.21.226:6783] error during connection attempt: dial tcp4 :0->10.50.21.226:6783: getsockopt: connection timed out\r\nINFO: 2017/06/18 14:06:57.825098 ->[10.50.21.91:6783] error during connection attempt: dial tcp4 :0->10.50.21.91:6783: getsockopt: connection timed out\r\nINFO: 2017/06/18 14:09:35.270454 ->[10.50.21.226:6783] attempting connection\r\nINFO: 2017/06/18 14:10:00.046305 ->[10.50.20.63:6783] attempting connection\r\nINFO: 2017/06/18 14:10:03.053167 ->[10.50.20.63:6783] error during connection attempt: dial tcp4 :0->10.50.20.63:6783: getsockopt: no route to host\r\nINFO: 2017/06/18 14:11:42.497042 ->[10.50.21.226:6783] error during connection attempt: dial tcp4 :0->10.50.21.226:6783: getsockopt: connection timed out\r\nINFO: 2017/06/18 14:11:49.102565 Expired MAC 0e:db:2c:c9:f4:e6 at b2:b0:8e:9e:92:01(ip-10-50-21-194.ec2.internal)\r\nINFO: 2017/06/18 14:11:49.102624 Expired MAC ba:b6:78:61:2d:9d at 12:4a:1a:49:54:d8(ip-10-50-20-214.ec2.internal)\r\n```\r\n\r\nIn the above, `10.50.21.91` and `10.50.21.226` and `10.50.20.63` are old nodes that have since been deleted from kube (not `NotReady` but actually deleted).\r\n\r\n```\r\n$ kubectl get nodes\r\nNAME           STATUS    AGE       VERSION\r\n10.50.20.144   Ready     1h        v1.6.2\r\n10.50.20.214   Ready     1h        v1.6.2\r\n10.50.21.194   Ready     1h        v1.6.2\r\n```\r\n\r\nI do not know if this is affecting some oddness with other pods, like services being unable to find endpoints even though they are fine, and sometimes nodes being unable to find the internal kube API service endpoint `10.200.0.1:443`. FWIW, `weave-kube` accesses it just fine:\r\n\r\n```\r\ncore@ip-10-50-20-214 ~ $ docker exec -it f7dcf35635b55 sh\r\n/home/weave # curl -ik https://10.200.0.1\r\nHTTP/1.1 403 Forbidden\r\nContent-Type: text/plain\r\nX-Content-Type-Options: nosniff\r\nDate: Sun, 18 Jun 2017 14:15:59 GMT\r\nContent-Length: 79\r\n\r\nUser \"system:anonymous\" cannot get  at the cluster scope.: \"No policy matched.\"\r\n/home/weave #\r\n```"}}
{"actor":{"display_login":"deitch"}}