{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/etcd-operator/issues/934","repository_url":"https://api.github.com/repos/coreos/etcd-operator","labels_url":"https://api.github.com/repos/coreos/etcd-operator/issues/934/labels{/name}","comments_url":"https://api.github.com/repos/coreos/etcd-operator/issues/934/comments","events_url":"https://api.github.com/repos/coreos/etcd-operator/issues/934/events","html_url":"https://github.com/coreos/etcd-operator/issues/934","id":219113470,"number":934,"title":"Failed to scale down the self hosted etcd clusters","user":{"login":"yifan-gu","id":3278588,"avatar_url":"https://avatars3.githubusercontent.com/u/3278588?v=3","gravatar_id":"","url":"https://api.github.com/users/yifan-gu","html_url":"https://github.com/yifan-gu","followers_url":"https://api.github.com/users/yifan-gu/followers","following_url":"https://api.github.com/users/yifan-gu/following{/other_user}","gists_url":"https://api.github.com/users/yifan-gu/gists{/gist_id}","starred_url":"https://api.github.com/users/yifan-gu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yifan-gu/subscriptions","organizations_url":"https://api.github.com/users/yifan-gu/orgs","repos_url":"https://api.github.com/users/yifan-gu/repos","events_url":"https://api.github.com/users/yifan-gu/events{/privacy}","received_events_url":"https://api.github.com/users/yifan-gu/received_events","type":"User","site_admin":false},"labels":[{"id":474142882,"url":"https://api.github.com/repos/coreos/etcd-operator/labels/kind/bug","name":"kind/bug","color":"ff0000","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-04-04T01:14:16Z","updated_at":"2017-04-04T01:18:41Z","closed_at":null,"body":"The pluton test for bootkube is trying to scale down from 3 to 1, and it often gets stuck.\r\nHere is some artifacts I got:\r\n\r\n**etcd cluster TPR object**:\r\n```\r\n$ kubectl get clusters -n kube-system -o yaml\r\napiVersion: v1\r\nitems:\r\n- apiVersion: etcd.coreos.com/v1beta1\r\n  kind: Cluster\r\n  metadata:\r\n    creationTimestamp: 2017-04-04T00:28:46Z\r\n    name: kube-etcd\r\n    namespace: kube-system\r\n    resourceVersion: \"1785\"\r\n    selfLink: /apis/etcd.coreos.com/v1beta1/namespaces/kube-system/clusters/kube-etcd\r\n    uid: aa4ecb7d-18cd-11e7-9d37-42010af0000e\r\n  spec:\r\n    TLS: null\r\n    pod:\r\n      antiAffinity: false\r\n      nodeSelector:\r\n        master: \"true\"\r\n      resources: {}\r\n    selfHosted:\r\n      bootMemberClientEndpoint: http://10.240.0.14:12379\r\n    size: 1\r\n    version: 3.1.0\r\n  status:\r\n    conditions:\r\n    - reason: 'Current cluster size: 0, desired cluster size: 1'\r\n      transitionTime: 2017-04-04T00:28:46.309813657Z\r\n      type: ScalingUp\r\n    - reason: \"\"\r\n      transitionTime: 2017-04-04T00:30:47.275086856Z\r\n      type: Ready\r\n    - reason: 'Current cluster size: 1, desired cluster size: 3'\r\n      transitionTime: 2017-04-04T00:33:42.513059964Z\r\n      type: ScalingUp\r\n    - reason: 'Current cluster size: 2, desired cluster size: 3'\r\n      transitionTime: 2017-04-04T00:36:00.546899788Z\r\n      type: ScalingUp\r\n    controlPaused: false\r\n    currentVersion: 3.1.0\r\n    members:\r\n      ready:\r\n      - kube-etcd-0000\r\n      - kube-etcd-0001\r\n    phase: Running\r\n    reason: \"\"\r\n    size: 3\r\n    targetVersion: \"\"\r\nkind: List\r\nmetadata: {}\r\nresourceVersion: \"\"\r\nselfLink: \"\"\r\n```\r\n\r\n**etcd operator logs**: [here](https://gist.github.com/yifan-gu/836b47963c2665bf57cf9d17e3841ce9)\r\n\r\n**kube-etcd logs**: [here](https://gist.github.com/yifan-gu/f6bda8a120082b9e5c014be3f280f856)\r\n\r\n@xiang90 @hongchaodeng \r\n\r\nI also keep the cluster running, so you can ssh into them and do more debug.\r\n\r\n```shell\r\ngcloud compute ssh yifan-bootkube-test-16f15d65500a0db42e11 --zone us-central1-a\r\ncd /home/core\r\nexport KUBECONFIG=/etc/kubernetes/kubeconfig\r\nsudo ./kubectl get nodes\r\n```"},"comment":{"url":"https://api.github.com/repos/coreos/etcd-operator/issues/comments/291361635","html_url":"https://github.com/coreos/etcd-operator/issues/934#issuecomment-291361635","issue_url":"https://api.github.com/repos/coreos/etcd-operator/issues/934","id":291361635,"user":{"login":"hongchaodeng","id":920884,"avatar_url":"https://avatars2.githubusercontent.com/u/920884?v=3","gravatar_id":"","url":"https://api.github.com/users/hongchaodeng","html_url":"https://github.com/hongchaodeng","followers_url":"https://api.github.com/users/hongchaodeng/followers","following_url":"https://api.github.com/users/hongchaodeng/following{/other_user}","gists_url":"https://api.github.com/users/hongchaodeng/gists{/gist_id}","starred_url":"https://api.github.com/users/hongchaodeng/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hongchaodeng/subscriptions","organizations_url":"https://api.github.com/users/hongchaodeng/orgs","repos_url":"https://api.github.com/users/hongchaodeng/repos","events_url":"https://api.github.com/users/hongchaodeng/events{/privacy}","received_events_url":"https://api.github.com/users/hongchaodeng/received_events","type":"User","site_admin":false},"created_at":"2017-04-04T01:18:41Z","updated_at":"2017-04-04T01:18:41Z","body":"@yifan-gu \r\n```\r\n$ gcloud compute ssh yifan-bootkube-test-16f15d65500a0db42e11 --zone us-central1-a\r\n\r\nERROR: (gcloud.compute.ssh) Could not fetch instance:\r\n - The resource 'projects/coreos-k8s-scale-testing/zones/us-central1-a/instances/yifan-bootkube-test-16f15d65500a0db42e11' was not found\r\n```"}}
{"actor":{"display_login":"hongchaodeng"}}