{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24322","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24322/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24322/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24322/events","html_url":"https://github.com/kubernetes/kubernetes/issues/24322","id":148578402,"number":24322,"title":"kube-proxy iptables: service will fail with fixed probability if one of endpoints unreachable","user":{"login":"njuicsgz","id":8842478,"avatar_url":"https://avatars1.githubusercontent.com/u/8842478?v=4","gravatar_id":"","url":"https://api.github.com/users/njuicsgz","html_url":"https://github.com/njuicsgz","followers_url":"https://api.github.com/users/njuicsgz/followers","following_url":"https://api.github.com/users/njuicsgz/following{/other_user}","gists_url":"https://api.github.com/users/njuicsgz/gists{/gist_id}","starred_url":"https://api.github.com/users/njuicsgz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/njuicsgz/subscriptions","organizations_url":"https://api.github.com/users/njuicsgz/orgs","repos_url":"https://api.github.com/users/njuicsgz/repos","events_url":"https://api.github.com/users/njuicsgz/events{/privacy}","received_events_url":"https://api.github.com/users/njuicsgz/received_events","type":"User","site_admin":false},"labels":[{"id":116712108,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/network","name":"sig/network","color":"d2b48c","default":false},{"id":173493827,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/team/cluster%20(deprecated%20-%20do%20not%20use)","name":"team/cluster (deprecated - do not use)","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2016-04-15T07:23:38Z","updated_at":"2017-12-16T11:16:38Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I am not sure this is a design issue or not, but it puzzled me a lot.\n\nI found my service fails with fixed probability if one of endpoints unreachable when use kube-proxy with iptables mode. \n\nIt is a critical problem that client's real requests will be refused when\n1) my service is down while container is running ok, before next readiness time interval come if I have readiness probe\n2) I am doing a rolling update\n\nThe user space mode does not have this problem since kube-proxy will make a real connection to endpoint before proxy the flow.\n\nSince AWS ELB could retry the next endpoint if one endpoint is unreachable, so should I hide my service behind of a ELB to avoid this issue? Bug how to do when I am in a baremetal env, create a nginx for each service?\n1. Environment:\n   root@ip-172-20-0-9:~# kubectl version\n   Client Version: version.Info{Major:\"1\", Minor:\"2\", GitVersion:\"v1.2.0\", GitCommit:\"5cb86ee022267586db386f62781338b0483733b3\", GitTreeState:\"clean\"}\n   Server Version: version.Info{Major:\"1\", Minor:\"2\", GitVersion:\"v1.2.0\", GitCommit:\"5cb86ee022267586db386f62781338b0483733b3\", GitTreeState:\"clean\"}\n2. How to reproduce:\n   2.1 create rc with 2 pods of nginx:\n   root@xx# cat nginx.rc.json \n   {\n   \"apiVersion\": \"v1\", \n   \"kind\": \"ReplicationController\", \n   \"metadata\": {\n       \"name\": \"nginx\", \n       \"namespace\": \"default\"\n   }, \n   \"spec\": {\n       \"replicas\": 2, \n       \"selector\": {\n           \"name\": \"nginx\", \n           \"version\": \"latest\"\n       }, \n       \"template\": {\n           \"metadata\": {\n               \"labels\": {\n                   \"name\": \"nginx\", \n                   \"version\": \"latest\"\n               }, \n               \"namespace\": \"default\"\n           }, \n           \"spec\": {\n               \"containers\": [\n                   {\n                       \"image\": \"nginx\", \n                       \"name\": \"nginx\",\n                       \"command\": [\n                           \"/bin/bash\",\n                           \"-c\",\n                           \"service nginx start && sleep 3600\"\n                       ],\n                       \"ports\": [], \n                       \"resources\": {\n                           \"limits\": {\n                               \"memory\": \"128Mi\"\n                           }\n                       }\n                   }\n               ]\n           }\n       }\n   }\n   }\n\nroot@ip-172-20-0-9:~# kubectl get po\nNAME          READY     STATUS    RESTARTS   AGE\nnginx-rbrob   1/1       Running   0          18m\nnginx-ucp14   1/1       Running   0          18m\n\n2.2 run a service\nroot@ip-172-20-0-9:~# kubectl describe service nginx\nName:                   nginx\nNamespace:              default\nLabels:                 <none>\nSelector:               name=nginx\nType:                   ClusterIP\nIP:                     10.0.253.87\nPort:                   <unset> 40080/TCP\nEndpoints:              10.244.1.4:80,10.244.1.5:80\nSession Affinity:       None\nNo events.\n\n2.3 disable one endpoint by close nginx service\nroot@ip-172-20-0-67:/home/ubuntu# docker exec -it 940c8b5b02b3 service nginx stop\nroot@ip-172-20-0-67:/home/ubuntu# docker exec -it 940c8b5b02b3 service nginx status\nnginx is not running ... failed!\n\n2.4 Curl the nginx by service IP, it fails in half chance.\nroot@ip-172-20-0-67:/home/ubuntu# curl -s -o /dev/null -w \"%{http_code}\" 10.0.253.87:40080\n200\nroot@ip-172-20-0-67:/home/ubuntu# curl -s -o /dev/null -w \"%{http_code}\" 10.0.253.87:40080\n000\nroot@ip-172-20-0-67:/home/ubuntu# curl -s -o /dev/null -w \"%{http_code}\" 10.0.253.87:40080\n000\nroot@ip-172-20-0-67:/home/ubuntu# curl -s -o /dev/null -w \"%{http_code}\" 10.0.253.87:40080\n200\n\n2.5 Curl ELB will return 200 always\nroot@ip-172-20-0-9:~# kubectl describe service nginx\nName:                   nginx\nNamespace:              default\nLabels:                 <none>\nSelector:               name=nginx\nType:                   LoadBalancer\nIP:                     10.0.145.10\nLoadBalancer Ingress:   a99110e7402db11e6978d0663f446fd2-1127146940.us-east-1.elb.amazonaws.com\nPort:                   <unset> 40080/TCP\nNodePort:               <unset> 32616/TCP\nEndpoints:              10.244.1.4:80,10.244.1.5:80\nSession Affinity:       None\nEvents:\n  FirstSeen     LastSeen        Count   From                    SubobjectPath   Type            Reason                  Message\n\n---\n\n  18m           18m             1       {service-controller }                   Normal          CreatingLoadBalancer    Creating load balancer\n  18m           18m             1       {service-controller }                   Normal          CreatedLoadBalancer     Created load balancer\n\n$ for((i=1;i<=1000;i++));do curl -s -o /dev/null -w \"%{http_code}\" a99110e7402db11e6978d0663f446fd2-1126940.us-east-1.elb.amazonaws.com:40080 && sleep 0.1; done\n200\n"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/352177044","html_url":"https://github.com/kubernetes/kubernetes/issues/24322#issuecomment-352177044","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/24322","id":352177044,"user":{"login":"fejta-bot","id":30488982,"avatar_url":"https://avatars1.githubusercontent.com/u/30488982?v=4","gravatar_id":"","url":"https://api.github.com/users/fejta-bot","html_url":"https://github.com/fejta-bot","followers_url":"https://api.github.com/users/fejta-bot/followers","following_url":"https://api.github.com/users/fejta-bot/following{/other_user}","gists_url":"https://api.github.com/users/fejta-bot/gists{/gist_id}","starred_url":"https://api.github.com/users/fejta-bot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fejta-bot/subscriptions","organizations_url":"https://api.github.com/users/fejta-bot/orgs","repos_url":"https://api.github.com/users/fejta-bot/repos","events_url":"https://api.github.com/users/fejta-bot/events{/privacy}","received_events_url":"https://api.github.com/users/fejta-bot/received_events","type":"User","site_admin":false},"created_at":"2017-12-16T11:16:38Z","updated_at":"2017-12-16T11:16:38Z","author_association":"NONE","body":"Issues go stale after 30d of inactivity.\nMark the issue as fresh with `/remove-lifecycle stale`.\nStale issues rot after an additional 30d of inactivity and eventually close.\n\nPrevent issues from auto-closing with an `/lifecycle frozen` comment.\n\nIf this issue is safe to close now please do so with `/close`.\n\nSend feedback to sig-testing, kubernetes/test-infra and/or `@fejta`.\n/lifecycle stale"}}
{"actor":{"display_login":"fejta-bot"}}