{"action":"created","issue":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/51369","repository_url":"https://api.github.com/repos/kubernetes/kubernetes","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/51369/labels{/name}","comments_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/51369/comments","events_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/51369/events","html_url":"https://github.com/kubernetes/kubernetes/pull/51369","id":253026448,"number":51369,"title":"kubeadm: Detect kubelet readiness and error out if the kubelet is unhealthy","user":{"login":"luxas","id":5442475,"avatar_url":"https://avatars1.githubusercontent.com/u/5442475?v=4","gravatar_id":"","url":"https://api.github.com/users/luxas","html_url":"https://github.com/luxas","followers_url":"https://api.github.com/users/luxas/followers","following_url":"https://api.github.com/users/luxas/following{/other_user}","gists_url":"https://api.github.com/users/luxas/gists{/gist_id}","starred_url":"https://api.github.com/users/luxas/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/luxas/subscriptions","organizations_url":"https://api.github.com/users/luxas/orgs","repos_url":"https://api.github.com/users/luxas/repos","events_url":"https://api.github.com/users/luxas/events{/privacy}","received_events_url":"https://api.github.com/users/luxas/received_events","type":"User","site_admin":false},"labels":[{"id":414883982,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/approved","name":"approved","color":"0ffa16","default":false},{"id":477397086,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/cncf-cla:%20yes","name":"cncf-cla: yes","color":"bfe5bf","default":false},{"id":148225179,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/lgtm","name":"lgtm","color":"15dd18","default":false},{"id":200149833,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/release-note","name":"release-note","color":"c2e0c6","default":false},{"id":173494222,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/sig/cluster-lifecycle","name":"sig/cluster-lifecycle","color":"d2b48c","default":false},{"id":253450978,"url":"https://api.github.com/repos/kubernetes/kubernetes/labels/size/L","name":"size/L","color":"ee9900","default":false}],"state":"open","locked":false,"assignee":{"login":"timothysc","id":169553,"avatar_url":"https://avatars0.githubusercontent.com/u/169553?v=4","gravatar_id":"","url":"https://api.github.com/users/timothysc","html_url":"https://github.com/timothysc","followers_url":"https://api.github.com/users/timothysc/followers","following_url":"https://api.github.com/users/timothysc/following{/other_user}","gists_url":"https://api.github.com/users/timothysc/gists{/gist_id}","starred_url":"https://api.github.com/users/timothysc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timothysc/subscriptions","organizations_url":"https://api.github.com/users/timothysc/orgs","repos_url":"https://api.github.com/users/timothysc/repos","events_url":"https://api.github.com/users/timothysc/events{/privacy}","received_events_url":"https://api.github.com/users/timothysc/received_events","type":"User","site_admin":false},"assignees":[{"login":"timothysc","id":169553,"avatar_url":"https://avatars0.githubusercontent.com/u/169553?v=4","gravatar_id":"","url":"https://api.github.com/users/timothysc","html_url":"https://github.com/timothysc","followers_url":"https://api.github.com/users/timothysc/followers","following_url":"https://api.github.com/users/timothysc/following{/other_user}","gists_url":"https://api.github.com/users/timothysc/gists{/gist_id}","starred_url":"https://api.github.com/users/timothysc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timothysc/subscriptions","organizations_url":"https://api.github.com/users/timothysc/orgs","repos_url":"https://api.github.com/users/timothysc/repos","events_url":"https://api.github.com/users/timothysc/events{/privacy}","received_events_url":"https://api.github.com/users/timothysc/received_events","type":"User","site_admin":false},{"login":"pipejakob","id":1605981,"avatar_url":"https://avatars2.githubusercontent.com/u/1605981?v=4","gravatar_id":"","url":"https://api.github.com/users/pipejakob","html_url":"https://github.com/pipejakob","followers_url":"https://api.github.com/users/pipejakob/followers","following_url":"https://api.github.com/users/pipejakob/following{/other_user}","gists_url":"https://api.github.com/users/pipejakob/gists{/gist_id}","starred_url":"https://api.github.com/users/pipejakob/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pipejakob/subscriptions","organizations_url":"https://api.github.com/users/pipejakob/orgs","repos_url":"https://api.github.com/users/pipejakob/repos","events_url":"https://api.github.com/users/pipejakob/events{/privacy}","received_events_url":"https://api.github.com/users/pipejakob/received_events","type":"User","site_admin":false},{"login":"mattmoyer","id":3210334,"avatar_url":"https://avatars3.githubusercontent.com/u/3210334?v=4","gravatar_id":"","url":"https://api.github.com/users/mattmoyer","html_url":"https://github.com/mattmoyer","followers_url":"https://api.github.com/users/mattmoyer/followers","following_url":"https://api.github.com/users/mattmoyer/following{/other_user}","gists_url":"https://api.github.com/users/mattmoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/mattmoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattmoyer/subscriptions","organizations_url":"https://api.github.com/users/mattmoyer/orgs","repos_url":"https://api.github.com/users/mattmoyer/repos","events_url":"https://api.github.com/users/mattmoyer/events{/privacy}","received_events_url":"https://api.github.com/users/mattmoyer/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/36","html_url":"https://github.com/kubernetes/kubernetes/milestone/36","labels_url":"https://api.github.com/repos/kubernetes/kubernetes/milestones/36/labels","id":2545392,"number":36,"title":"v1.8","description":"Kubernetes 1.8 release milestone, date is TBD.","creator":{"login":"timothysc","id":169553,"avatar_url":"https://avatars0.githubusercontent.com/u/169553?v=4","gravatar_id":"","url":"https://api.github.com/users/timothysc","html_url":"https://github.com/timothysc","followers_url":"https://api.github.com/users/timothysc/followers","following_url":"https://api.github.com/users/timothysc/following{/other_user}","gists_url":"https://api.github.com/users/timothysc/gists{/gist_id}","starred_url":"https://api.github.com/users/timothysc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timothysc/subscriptions","organizations_url":"https://api.github.com/users/timothysc/orgs","repos_url":"https://api.github.com/users/timothysc/repos","events_url":"https://api.github.com/users/timothysc/events{/privacy}","received_events_url":"https://api.github.com/users/timothysc/received_events","type":"User","site_admin":false},"open_issues":204,"closed_issues":371,"state":"open","created_at":"2017-05-26T17:18:43Z","updated_at":"2017-09-03T15:46:31Z","due_on":null,"closed_at":null},"comments":16,"created_at":"2017-08-25T21:51:32Z","updated_at":"2017-09-03T16:38:51Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/kubernetes/kubernetes/pulls/51369","html_url":"https://github.com/kubernetes/kubernetes/pull/51369","diff_url":"https://github.com/kubernetes/kubernetes/pull/51369.diff","patch_url":"https://github.com/kubernetes/kubernetes/pull/51369.patch"},"body":"<!--  Thanks for sending a pull request!  Here are some tips for you:\r\n1. If this is your first time, read our contributor guidelines https://git.k8s.io/community/contributors/devel/pull-requests.md#the-pr-submit-process and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide\r\n2. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/devel/pull-requests.md#best-practices-for-faster-reviews\r\n3. Follow the instructions for writing a release note: https://git.k8s.io/community/contributors/devel/pull-requests.md#write-release-notes-if-needed\r\n-->\r\n\r\n**What this PR does / why we need it**:\r\n\r\nIn order to improve the UX when the kubelet is unhealthy or stopped, or whatever, kubeadm now polls the kubelet's API after 40 and 60 seconds, and then performs an exponential backoff for a total of 155 seconds.\r\n\r\nIf the kubelet endpoint is not returning `ok` by then, kubeadm gives up and exits.\r\n\r\nThis will miligate at least 60% of our \"[apiclient] Created API client, waiting for control plane to come up\" issues in the kubeadm issue tracker :tada:, as kubeadm now informs the user what's wrong and also doesn't deadlock like before.\r\n\r\nDemo:\r\n```\r\nlucas@THEGOPHER:~/luxas/kubernetes$ sudo ./kubeadm init --skip-preflight-checks\r\n[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.\r\n[init] Using Kubernetes version: v1.7.4\r\n[init] Using Authorization modes: [Node RBAC]\r\n[preflight] Skipping pre-flight checks\r\n[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)\r\n[certificates] Generated ca certificate and key.\r\n[certificates] Generated apiserver certificate and key.\r\n[certificates] apiserver serving cert is signed for DNS names [thegopher kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.115]\r\n[certificates] Generated apiserver-kubelet-client certificate and key.\r\n[certificates] Generated sa key and public key.\r\n[certificates] Generated front-proxy-ca certificate and key.\r\n[certificates] Generated front-proxy-client certificate and key.\r\n[certificates] Valid certificates and keys now exist in \"/etc/kubernetes/pki\"\r\n[kubeconfig] Wrote KubeConfig file to disk: \"admin.conf\"\r\n[kubeconfig] Wrote KubeConfig file to disk: \"kubelet.conf\"\r\n[kubeconfig] Wrote KubeConfig file to disk: \"controller-manager.conf\"\r\n[kubeconfig] Wrote KubeConfig file to disk: \"scheduler.conf\"\r\n[controlplane] Wrote Static Pod manifest for component kube-apiserver to \"/etc/kubernetes/manifests/kube-apiserver.yaml\"\r\n[controlplane] Wrote Static Pod manifest for component kube-controller-manager to \"/etc/kubernetes/manifests/kube-controller-manager.yaml\"\r\n[controlplane] Wrote Static Pod manifest for component kube-scheduler to \"/etc/kubernetes/manifests/kube-scheduler.yaml\"\r\n[etcd] Wrote Static Pod manifest for a local etcd instance to \"/etc/kubernetes/manifests/etcd.yaml\"\r\n[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory \"/etc/kubernetes/manifests\"\r\n[init] This often takes around a minute; or longer if the control plane images have to be pulled.\r\n[apiclient] All control plane components are healthy after 40.502199 seconds\r\n[markmaster] Will mark node thegopher as master by adding a label and a taint\r\n[markmaster] Master thegopher tainted and labelled with key/value: node-role.kubernetes.io/master=\"\"\r\n[bootstraptoken] Using token: 5776d5.91e7ed14f9e274df\r\n[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials\r\n[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token\r\n[bootstraptoken] Creating the \"cluster-info\" ConfigMap in the \"kube-public\" namespace\r\n[uploadconfig] Storing the configuration used in ConfigMap \"kubeadm-config\" in the \"kube-system\" Namespace\r\n[addons] Applied essential addon: kube-dns\r\n[addons] Applied essential addon: kube-proxy\r\n\r\nYour Kubernetes master has initialized successfully!\r\n\r\nTo start using your cluster, you need to run (as a regular user):\r\n\r\n  mkdir -p $HOME/.kube\r\n  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config\r\n  sudo chown $(id -u):$(id -g) $HOME/.kube/config\r\n\r\nYou should now deploy a pod network to the cluster.\r\nRun \"kubectl apply -f [podnetwork].yaml\" with one of the options listed at:\r\n  http://kubernetes.io/docs/admin/addons/\r\n\r\nYou can now join any number of machines by running the following on each node\r\nas root:\r\n\r\n  kubeadm join --token 5776d5.91e7ed14f9e274df 192.168.1.115:6443 --discovery-token-ca-cert-hash sha256:6f301ce8c3f5f6558090b2c3599d26d6fc94ffa3c3565ffac952f4f0c7a9b2a9\r\n\r\nlucas@THEGOPHER:~/luxas/kubernetes$ sudo ./kubeadm reset\r\n[preflight] Running pre-flight checks\r\n[reset] Stopping the kubelet service\r\n[reset] Unmounting mounted directories in \"/var/lib/kubelet\"\r\n[reset] Removing kubernetes-managed containers\r\n[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes /var/lib/etcd]\r\n[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]\r\n[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]\r\nlucas@THEGOPHER:~/luxas/kubernetes$ sudo systemctl stop kubelet\r\nlucas@THEGOPHER:~/luxas/kubernetes$ sudo ./kubeadm init --skip-preflight-checks\r\n[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.\r\n[init] Using Kubernetes version: v1.7.4\r\n[init] Using Authorization modes: [Node RBAC]\r\n[preflight] Skipping pre-flight checks\r\n[kubeadm] WARNING: starting in 1.8, tokens expire after 24 hours by default (if you require a non-expiring token use --token-ttl 0)\r\n[certificates] Generated ca certificate and key.\r\n[certificates] Generated apiserver certificate and key.\r\n[certificates] apiserver serving cert is signed for DNS names [thegopher kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.1.115]\r\n[certificates] Generated apiserver-kubelet-client certificate and key.\r\n[certificates] Generated sa key and public key.\r\n[certificates] Generated front-proxy-ca certificate and key.\r\n[certificates] Generated front-proxy-client certificate and key.\r\n[certificates] Valid certificates and keys now exist in \"/etc/kubernetes/pki\"\r\n[kubeconfig] Wrote KubeConfig file to disk: \"admin.conf\"\r\n[kubeconfig] Wrote KubeConfig file to disk: \"kubelet.conf\"\r\n[kubeconfig] Wrote KubeConfig file to disk: \"controller-manager.conf\"\r\n[kubeconfig] Wrote KubeConfig file to disk: \"scheduler.conf\"\r\n[controlplane] Wrote Static Pod manifest for component kube-apiserver to \"/etc/kubernetes/manifests/kube-apiserver.yaml\"\r\n[controlplane] Wrote Static Pod manifest for component kube-controller-manager to \"/etc/kubernetes/manifests/kube-controller-manager.yaml\"\r\n[controlplane] Wrote Static Pod manifest for component kube-scheduler to \"/etc/kubernetes/manifests/kube-scheduler.yaml\"\r\n[etcd] Wrote Static Pod manifest for a local etcd instance to \"/etc/kubernetes/manifests/etcd.yaml\"\r\n[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory \"/etc/kubernetes/manifests\"\r\n[init] This often takes around a minute; or longer if the control plane images have to be pulled.\r\n[kubelet-check] It seems like the kubelet isn't running or healthy.\r\n[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r\n[kubelet-check] It seems like the kubelet isn't running or healthy.\r\n[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r\n[kubelet-check] It seems like the kubelet isn't running or healthy.\r\n[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r\n[kubelet-check] It seems like the kubelet isn't running or healthy.\r\n[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r\n[kubelet-check] It seems like the kubelet isn't running or healthy.\r\n[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r\n[kubelet-check] It seems like the kubelet isn't running or healthy.\r\n[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r\n[kubelet-check] It seems like the kubelet isn't running or healthy.\r\n[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r\n[kubelet-check] It seems like the kubelet isn't running or healthy.\r\n[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz/syncloop' failed with error: Get http://localhost:10255/healthz/syncloop: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r\n[kubelet-check] It seems like the kubelet isn't running or healthy.\r\n[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10255/healthz' failed with error: Get http://localhost:10255/healthz: dial tcp 127.0.0.1:10255: getsockopt: connection refused.\r\n\r\nUnfortunately, an error has occurred:\r\n\ttimed out waiting for the condition\r\n\r\nThis error is likely caused by that:\r\n\t- The kubelet is not running\r\n\t- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)\r\n\t- There is no internet connection; so the kubelet can't pull the following control plane images:\r\n\t\t- gcr.io/google_containers/kube-apiserver-amd64:v1.7.4\r\n\t\t- gcr.io/google_containers/kube-controller-manager-amd64:v1.7.4\r\n\t\t- gcr.io/google_containers/kube-scheduler-amd64:v1.7.4\r\n\r\nYou can troubleshoot this for example with the following commands if you're on a systemd-powered system:\r\n\t- 'systemctl status kubelet'\r\n\t- 'journalctl -xeu kubelet'\r\ncouldn't initialize a Kubernetes cluster\r\n```\r\n\r\nIn this demo, I'm first starting kubeadm normally and everything works as usual.\r\nIn the second case, I'm explicitely stopping the kubelet so it doesn't run, and skipping preflight checks, so that kubeadm doesn't even try to exec `systemctl start kubelet` like it does usually.\r\nThat obviously results in a non-working system, but now kubeadm tells the user what's the problem instead of waiting forever.\r\n\r\n**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #\r\n\r\nFixes: https://github.com/kubernetes/kubeadm/issues/377\r\n\r\n**Special notes for your reviewer**:\r\n\r\n**Release note**:\r\n<!--  Steps to write your release note:\r\n1. Use the release-note-* labels to set the release note state (if you have access)\r\n2. Enter your extended release note in the below block; leaving it blank means using the PR title as the release note. If no release note is required, just write `NONE`.\r\n-->\r\n```release-note\r\nkubeadm: Detect kubelet readiness and error out if the kubelet is unhealthy\r\n```\r\n@kubernetes/sig-cluster-lifecycle-pr-reviews @pipejakob \r\n\r\ncc @justinsb @kris-nova @lukemarsden as well as you wanted this feature :)"},"comment":{"url":"https://api.github.com/repos/kubernetes/kubernetes/issues/comments/326815739","html_url":"https://github.com/kubernetes/kubernetes/pull/51369#issuecomment-326815739","issue_url":"https://api.github.com/repos/kubernetes/kubernetes/issues/51369","id":326815739,"user":{"login":"k8s-ci-robot","id":20407524,"avatar_url":"https://avatars0.githubusercontent.com/u/20407524?v=4","gravatar_id":"","url":"https://api.github.com/users/k8s-ci-robot","html_url":"https://github.com/k8s-ci-robot","followers_url":"https://api.github.com/users/k8s-ci-robot/followers","following_url":"https://api.github.com/users/k8s-ci-robot/following{/other_user}","gists_url":"https://api.github.com/users/k8s-ci-robot/gists{/gist_id}","starred_url":"https://api.github.com/users/k8s-ci-robot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/k8s-ci-robot/subscriptions","organizations_url":"https://api.github.com/users/k8s-ci-robot/orgs","repos_url":"https://api.github.com/users/k8s-ci-robot/repos","events_url":"https://api.github.com/users/k8s-ci-robot/events{/privacy}","received_events_url":"https://api.github.com/users/k8s-ci-robot/received_events","type":"User","site_admin":false},"created_at":"2017-09-03T16:38:51Z","updated_at":"2017-09-03T16:38:51Z","author_association":"COLLABORATOR","body":"@luxas: The following test **failed**, say `/retest` to rerun them all:\n\nTest name | Commit | Details | Rerun command\n--- | --- | --- | ---\npull-kubernetes-e2e-kops-aws | 92c5997b8ea910fa27925443d4f9fd6420e0c889 | [link](https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/51369/pull-kubernetes-e2e-kops-aws/46424/) | `/test pull-kubernetes-e2e-kops-aws`\n\n[Full PR test history](https://k8s-gubernator.appspot.com/pr/51369). [Your PR dashboard](https://k8s-gubernator.appspot.com/pr/luxas). Please help us cut down on flakes by [linking to](https://github.com/kubernetes/community/blob/master/contributors/devel/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes/kubernetes/issues?q=is:issue+is:open) when you hit one in your PR.\n\n<details>\n\nInstructions for interacting with me using PR comments are available [here](https://github.com/kubernetes/community/blob/master/contributors/devel/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://github.com/kubernetes/test-infra/blob/master/commands.md).\n</details>\n<!-- test report -->"}}
{"actor":{"display_login":"k8s-ci-robot"}}