{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/48705","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/48705/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/48705/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/48705/events","html_url":"https://github.com/kubernetes/kubernetes/issues/48705","id":241713359,"number":48705,"title":"Kube proxy fails and then hangs without dying.","user":{"login":"krish7919","id":5066922,"avatar_url":"https://avatars0.githubusercontent.com/u/5066922?v=4","gravatar_id":"","url":"https://api.github.com/users/krish7919","html_url":"https://github.com/krish7919","followers_url":"https://api.github.com/users/krish7919/followers","following_url":"https://api.github.com/users/krish7919/following{/other_user}","gists_url":"https://api.github.com/users/krish7919/gists{/gist_id}","starred_url":"https://api.github.com/users/krish7919/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krish7919/subscriptions","organizations_url":"https://api.github.com/users/krish7919/orgs","repos_url":"https://api.github.com/users/krish7919/repos","events_url":"https://api.github.com/users/krish7919/events{/privacy}","received_events_url":"https://api.github.com/users/krish7919/received_events","type":"User","site_admin":false},"labels":[{"id":128716589,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/area/kube-proxy","name":"area/kube-proxy","color":"0052cc","default":false},{"id":105146071,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/kind/bug","name":"kind/bug","color":"e11d21","default":false},{"id":621158926,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/azure","name":"sig/azure","color":"d2b48c","default":false},{"id":116712108,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/network","name":"sig/network","color":"d2b48c","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2017-07-10T13:32:57Z","updated_at":"2017-07-20T08:48:47Z","closed_at":null,"body":"<!-- This form is for bug reports and feature requests ONLY! \r\n\r\nIf you're looking for help check [Stack Overflow](https://stackoverflow.com/questions/tagged/kubernetes) and the [troubleshooting guide](https://kubernetes.io/docs/tasks/debug-application-cluster/troubleshooting/).\r\n-->\r\n\r\n**Is this a BUG REPORT or FEATURE REQUEST?**:\r\n\r\n> Uncomment only one, leave it on its own line: \r\n>\r\n/kind bug\r\n> /kind feature\r\n\r\n\r\n**What happened**:\r\nService was not able to communicate with backend deployment.\r\nChecked the logs and found that kube-proxy had panic-ed.\r\nMy assumption is it was because the dependencies of kube-proxy did not spin up before kube-proxy itself.\r\n\r\n**What you expected to happen**:\r\nExpectation is that kube-proxy would panic in this case and restart/recover.\r\nIt recovered after a restart of the kube-proxy on all the agents.\r\n\r\n**How to reproduce it (as minimally and precisely as possible)**:\r\nSpin up a new Kubernetes cluster on Azure container Service.\r\nI did it twice and it happened both the times.\r\n\r\n**Anything else we need to know?**:\r\nFix is to restart the kube-proxy. I think it is dependency management related. Stacktrace:\r\n\r\n```\r\n\r\n\r\ndocker logs  k8s_kube-proxy_kube-proxy-lhsw2_kube-system_985fbe1c-6193-11e7-af56-000d3a\r\n2666d5_0 \r\nI0705 15:07:02.183142       1 server.go:225] Using iptables Proxier.\r\nI0705 15:07:02.244796       1 server.go:249] Tearing down userspace rules.\r\nI0705 15:07:02.362487       1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072\r\nI0705 15:07:02.362993       1 conntrack.go:66] Setting conntrack hashsize to 32768\r\nI0705 15:07:02.363239       1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400\r\nI0705 15:07:02.363267       1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600\r\nI0705 15:40:08.124291       1 logs.go:41] http: panic serving 168.63.129.16:58960: runtime error: invalid memory address or nil pointer dereference\r\ngoroutine 962 [running]:\r\nnet/http.(*conn).serve.func1(0xc4202a4d00)\r\n        /usr/local/go/src/net/http/server.go:1491 +0x12a\r\npanic(0x3a4c580, 0xc420016040)\r\n        /usr/local/go/src/runtime/panic.go:458 +0x243\r\nk8s.io/kubernetes/pkg/proxy/healthcheck.hcHandler.ServeHTTP(0xc4209d28c7, 0x7, 0xc4209d28c0, 0x7, 0xc420103860, 0x6cd0ea0, 0xc420d080\r\nd0, 0xc420fec690)\r\n        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/proxy/healthcheck/healthcheck.go:197 +0x7c\r\nk8s.io/kubernetes/pkg/proxy/healthcheck.(*hcHandler).ServeHTTP(0xc420c68ff0, 0x6cd0ea0, 0xc420d080d0, 0xc420fec690)\r\n        <autogenerated>:8 +0x8c\r\nnet/http.serverHandler.ServeHTTP(0xc42041c080, 0x6cd0ea0, 0xc420d080d0, 0xc420fec690)\r\n        /usr/local/go/src/net/http/server.go:2202 +0x7d\r\nnet/http.(*conn).serve(0xc4202a4d00, 0x6cd3fa0, 0xc4208a8400)\r\n        /usr/local/go/src/net/http/server.go:1579 +0x4b7\r\ncreated by net/http.(*Server).Serve\r\n        /usr/local/go/src/net/http/server.go:2293 +0x44d\r\n```\r\n\r\n**Environment**:\r\n- Kubernetes version (use `kubectl version`):\r\n```\r\nkubectl version\r\nClient Version: version.Info{Major:\"1\", Minor:\"7\", GitVersion:\"v1.7.0\", GitCommit:\"d3ada0119e776222f11ec7945e6d860061339aad\", GitTreeState:\"clean\", BuildDate:\"2017-06-29T23:15:59Z\", GoVersion:\"go1.8.3\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"6\", GitVersion:\"v1.6.6\", GitCommit:\"7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\", GitTreeState:\"clean\", BuildDate:\"2017-06-16T18:21:54Z\", GoVersion:\"go1.7.6\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n- Cloud provider or hardware configuration**:\r\nAzure Container Service\r\n\r\n- OS (e.g. from /etc/os-release):\r\n```\r\ncat /etc/os-release \r\nNAME=\"Ubuntu\"\r\nVERSION=\"16.04.2 LTS (Xenial Xerus)\"\r\nID=ubuntu\r\nID_LIKE=debian\r\nPRETTY_NAME=\"Ubuntu 16.04.2 LTS\"\r\nVERSION_ID=\"16.04\"\r\nHOME_URL=\"http://www.ubuntu.com/\"\r\nSUPPORT_URL=\"http://help.ubuntu.com/\"\r\nBUG_REPORT_URL=\"http://bugs.launchpad.net/ubuntu/\"\r\nVERSION_CODENAME=xenial\r\nUBUNTU_CODENAME=xenial\r\n```\r\n\r\n- Kernel (e.g. `uname -a`):\r\n```\r\n uname -a\r\nLinux k8s-master-DA6EDAE7-0 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux\r\n```\r\n\r\n- Install tools:\r\n- Others:\r\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/316639294","html_url":"https://github.com/kubernetes/kubernetes/issues/48705#issuecomment-316639294","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/48705","id":316639294,"user":{"login":"krish7919","id":5066922,"avatar_url":"https://avatars0.githubusercontent.com/u/5066922?v=4","gravatar_id":"","url":"https://api.github.com/users/krish7919","html_url":"https://github.com/krish7919","followers_url":"https://api.github.com/users/krish7919/followers","following_url":"https://api.github.com/users/krish7919/following{/other_user}","gists_url":"https://api.github.com/users/krish7919/gists{/gist_id}","starred_url":"https://api.github.com/users/krish7919/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krish7919/subscriptions","organizations_url":"https://api.github.com/users/krish7919/orgs","repos_url":"https://api.github.com/users/krish7919/repos","events_url":"https://api.github.com/users/krish7919/events{/privacy}","received_events_url":"https://api.github.com/users/krish7919/received_events","type":"User","site_admin":false},"created_at":"2017-07-20T08:48:47Z","updated_at":"2017-07-20T08:48:47Z","body":"@caseydavenport The commit above does fix the issue. However, I have not personally tested it as I run on Azure managed Kubernetes and it hasn't updated to v1.7 yet.\r\n\r\nIs there any way the Azure team can update us on this?"}}
{"actor":{"display_login":"krish7919"}}